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PREFACE 



In 1994, the National Center for Education Statistics (NCES) published the Quality Profile for 
SASS, Aspects of the Quality of Data in the Schools and Staffing Surveys (SASS) (Jabine, 1994). 
The report presented and summarized the available information about the quality of data from 
the five surveys that comprised SASS during its first two rounds of data collection (1987-88 and 
1990-91). 

The present report, the second edition of the Quality Profile for SASS, updates the information 
from the earlier report and incorporates information about the quality of data from the various 
surveys that comprised SASS during its third round of data collection. The report also presents 
background on the survey design and procedures for each of the surveys. This report was 
prepared by Westat under contract to the National Center for Education Statistics, Contract No. 
RN 94093001, Task 30. 

The current Quality Profile was prepared by Graham Kalton, Marianne Winglee, Sheila 
Krawchuk, and Daniel Levine of Westat. Significant contributions to the second edition were 
made by selected staff of both NCES and the U.S. Census Bureau. Specifically, at NCES, 
Steven Kaufman served as project coordinator, providing counsel, direction, and support to the 
effort. Kerry Gruber and Dan Kasprzyk provided relevant source materials and clarified the 
many technical issues that arose during the preparation of this report. 

Since the main survey data collection and processing operations for SASS are carried out by the 
U.S. Census Bureau under an interagency agreement, selected source materials and the answers 
to a number of technical questions were obtained from members of the census staff, especially 
Dennis Schwanz, Randall Parmer, and Sharon Fondelief. 

Peer reviewers for this report were Ellen Bradbum and Kathryn Chandler, Early Childhood, 
International and Crosscutting Studies Division, NCES; Beth Young, Elementary/Secondary and 
Libraries Studies Division, NCES; Dennis Schwanz, Census Bureau; Leslie Scott, Education 
Statistics Services Institute; and Thomas B. Jabine, statistical consultant. Marilyn McMillen, 
Statistical Standards Program, NCES, was the adjudicator for the peer review. The authors are 
indebted to those who contributed to this effort as reviewers and provided many, valuable 
comments and suggestions. 

As additional SASS data collection rounds are conducted, NCES plans to issue updated editions 
of the Quality Profile. Accordingly, comments and suggestions on the format and content are 
welcome. 
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1. INTRODUCTION AND OVERVIEW 



1.1 INTRODUCTION 

The Schools and Staffing Surveys (SASS) is an integrated system of periodic sample surveys 
providing information, about teachers and administrators and the general condition of America's 
public and private elementary and secondary schools. Sponsored by the National Center for 
Education Statistics (NCES) of the U.S. Department of Education, SASS offers a source of data 
for policymakers, educators, education researchers, and the public. 

NCES initiated SASS in the mid-1980s in response to needs for information about critical 
aspects of teacher supply and demand, the qualifications and working conditions of teachers and 
principals, and the basic conditions in schools as workplaces and learning environments. SASS 
has been conducted three times: Round 1 in 1987-88, Round 2 in 1990-91, and Round 3 in 
1993-94. Round 4 is being fielded in the 1999-00 school year. At each round, NCES reviews 
the SASS content to expand, retain, or delete topics covered in the previous administration, 
maintaining the survey's capability for trend analysis, and adding new topics to address current 
concerns. 

SASS is an integrated system of several surveys. The survey data are collected by mail, with 
telephone followup of nonrespondents. . In the first two rounds, SASS comprised five 
components: the School Survey, the School Administrator Survey (now known as the School 
Principal Survey), the Teacher Demand and Shortage Survey (TDSS), the Teacher Survey, and 
the Teacher Followup Survey that was conducted the year after the core surveys. In Round 3, 
SASS added the Library Media Center Survey, the Library Media Specialist/Librarian Survey, 
and the Student Records Survey resulting in a system of eight surveys in total. Round 4 
administers six of these surveys, excluding the Library Media Specialist/Librarian Survey and 
the Student Records Survey. 

Purpose and audience. The purpose of A Quality Profile for SASS is to summarize what is 
known about the quality of data from the surveys that comprise SASS and to provide information 
about the survey design and procedures for each survey. Without such a summary, anyone 
wanting information about the quality of SASS estimates would have to search through a large 
body of literature, some of it not easily accessible. The Quality Profile draws on that literature 
and provides references for readers who want more detailed information. 

This report will be of interest to users of SASS data, to persons responsible for various aspects of 
the design and operation of the surveys, and to anyone interested in the quality of survey data, 
especially data from mail surveys and surveys related to education. More specifically, the report 
provides the basis for a systematic review of past and ongoing research on the quality of SASS 
data, with a view toward identifying gaps in our knowledge and establishing priorities for future 
research activities. 

Scope and structure of the report. This, the second edition of A Quality Profile for SASS, is a 
revision of the first edition (Jabine, 1994) that described the design and procedures used in 
Round 2, along with information on major changes between Rounds 1 and 2. This edition 
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expands on the first edition and discusses the progress and development of SASS from Rounds 1 
through 3. Since Round 4 had not been completed at the time that this report was prepared, the 
report does not cover that round. However, some new features introduced in Round 4 are 
mentioned in relevant chapters. 

There are several possible methods of organizing data on errors in surveys. The method used in 
this report follows the structure adopted in the first edition of the Quality Profile for SASS, 
namely presenting information on errors associated with each phase of survey operations: frame 
design and maintenance, sample selection, data collection, data processing, estimation, and 
evaluation of survey estimates. The report consists of nine chapters. Chapters 2 to 8 use this 
approach to describe, in turn, the quality of each of the SASS component surveys. 

■ The School Survey (chapter 2) is the core survey providing basic data on a sample of 
elementary and secondary schools, and it also serves as the sampling frame for the other 
surveys in SASS. It collects data on characteristics of both public and private schools, such 
as enrollment, student-staff ratios, programs and services offered, and length of the school 
day. 

■ The School Principal Survey (chapter 3) collects information about the demographic 
characteristics, training, experience, and salary of school heads/principals, and their 
judgments about the seriousness of school problems. This survey was known as the School 
Administrator Survey in Rounds 1 and 2. Its name was changed to the School Principal 
Survey in Round 3. For simplicity, it is referred to as the School Principal Survey throughout 
this report. 

■ The Teacher Demand and Shortage Survey (chapter 4) gathers data on teacher recruitment, 
hiring practices, teacher dismissals, existence of a teacher union, length of the contract year, 
teacher salary schedules, school choice, magnet programs, graduation requirements, and 
professional development for teachers and administrators. Information for public schools is 
obtained from the local educational agencies (LEAs), whereas private schools and BIA 
schools which are not part of an LEA are contacted directly. This survey is known as the 
School District Survey in Round 4. 

■ The Teacher Survey (chapter 5) collects data from teachers about their education, training, 
teaching assignment, certification, workload, and perceptions and attitudes about teaching. 

■ The Teacher Followup Survey (chapter 6) is conducted about 1 year after the Teacher Survey 
to measure teacher retention, naobility, and attrition from the profession in the intervening 
year. 

■ The Library Media Center and Library Media Specialist/Librarian Surveys (chapter 7). The 
data from the Library Media Center Survey provide a national picture of school libraries and 
permit an assessment of the adequacy of school libraries to meet the needs of students and 
staff. The separate Library Media Specialist/Librarian Survey collects data about school 
librarians, their educational background, work experience, and demographic characteristics, 
as well as duties, salaries, workload, and attitudes about their current position and their 
profession. 
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E3 The Student Records Survey (chapter 8) collects data that can be used to examine the 
distribution of school programs and teachers among students of differing demographic and 
academic characteristics and to describe the participation of students in school programs and 
services. 

The topics discussed in chapters 2 through 8 for each of the surveys include potential sources of 
errors — noncoverage, unit and item nonresponse, measurement and sampling errors — and their 
possible impact on the accuracy of the survey estimates. Chapter 2 provides the most detail on 
those sample design and data collection procedures used in all the SASS surveys. Subsequent 
chapters often refer back to chapter 2 for these details. 

Chapter 9 looks at SASS as a whole and broadens the discussion of quality to cover issues of 
relevance, accessibility, timeliness, and periodicity. This chapter also attempts to combine the 
findings from earlier chapters to identify key areas where efforts for methodological 
improvements might be most effectively directed and where further information is needed for the 
assessment of survey quality. 

1.2 AN OVERVIEW OF SASS 

Objectives. SASS is designed to provide periodic, timely data on public and private elementary 
and secondary schools in the United States. The analytical power of the data is enhanced by the 
ability to link survey data for individual LEAs, schools, teachers, and all other SASS 
components. The use of comparable questions in each round of SASS makes it possible to 
monitor changes in the nation's elementary and secondary education system. In each round, 
special inquiries can be included, subject to constraints on overall cost and burden on 
respondents. 

In Rounds 1 and 2, SASS data provided a basis for addressing five major policy issues (Hudson 
and Darling-Hammond, 1987): 

■ Teacher supply and demand. In what teaching fields do shortages exist? What school 
characteristics influence teacher supply and demand? How do the characteristics of new 
hires compare with those of the existing workforce? What distinguishes teachers who leave 
the profession from those who stay in it? What incentives are used to recruit and retain 
teachers in areas of shortage? 

B Characteristics of elementary and secondary teachers. How does the training and 
experience of teachers compare for different types of schools? How does teacher training 
relate to their fields of assignment? What are the characteristics of specific subgroups of 
teachers, such as bilingual teachers? 

0 Teacher workplace conditions. How are teachers affected by working conditions, including 
teaching workloads, student-teacher ratios, and resources available for teaching and 
professional development? How do teachers evaluate their working conditions? 
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■ Characteristics of school principals. What education and experience do the nation's 
elementary and secondary school principals have? What problems do they consider to be 
serious, and how do they evaluate their influence on school policies? 

■ School programs and policies. How do schools vary with respect to admission requirements, 
graduation requirements, teacher salaries and benefits, teaching load, and staffing patterns? 
How do graduation and college application rates vary by school? How many schools have 
special programs, such as remedial reading and mathematics, programs for the disabled, 
programs for the gifted and talented, and extended daycare programs? How many students 
are served by these programs? 

Round 3 added three new surveys to address policy issues regarding: 

■ Student participation in school programs and services. How are school programs and 
teachers distributed among students of differing demographic and academic characteristics? 

■ Library facilities and librarians. Are school libraries adequately equipped to meet the needs 
of students and teachers? What are the qualifications, experiences, and workloads of 
librarians? 

In Round 4, the content of SASS emphasizes the measurement of conditions and practices 
related to school reform (NCES, 1999). New issues being examined in Round 4 include the 
following: 

■ School practices. What are the structure and processes of school organization, management, 
and decisionmaking, including the degree Of authority and autonomy that teachers 
experience, and the influence of administrators, staff, teachers, and school site councils (or 
other decisionmaking bodies) on school policy and practice? 

B Parental involvement. What are the school practices that encourage and support parental 
involvenient in schools and build parenting skills related to education? 

B School safety. How do principals and teachers view school safety and discipline problems? 
What measures do schools take to prevent and remediate them? 

B Charter schools. What are the ; characteristics of charter schools? When were schools 
granted charter status and by whom? What types of regulations were waived to establish the 
schools? How many schools are new or converted from pre-existing schools? How many 
schools operate within a school district? 

■ 

Evolution and changes in the SASS design. Round 1 of SASS in 1987-88 integrated the design 
and operations of three existing survey programs: the Public and Private School Surveys, the 
Teacher Surveys, and the TDSSs. Prior to initiating SASS, a Public School Survey, covering 
both schools and teachers, had been conducted for school year 1984-85, and Private School 
Surveys, which included teachers, had been undertaken in 1983-84 and 1985-86. Separate 
TDSSs, covering LEAs and private schools, had been conducted for school years 1978-79 and 
1983-84 (NCES, 1992). 
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Round 2. There were significant changes between Rounds 1 and 2 of SASS in the sampling 
frames from which samples of public and private schools were selected and in the sample design 
for schools and teachers (see section 2.4). In Round 1, the primary frame for each sector was a 
list of schools purchased from Quality Education Data, Inc. (the QED list). For the private 
sector, the QED list was supplemented with lists obtained from several private school 
associations and by an area sample. In Round 2, the main public school frame for SASS was the 
list of schools developed from NGES's 1988-89 Common Core of Data (CCD), which included 
an annual census of LEAs and schools. For private schools, the 1988-89 QED list, supplemented 
by lists obtained from private school associations, was used as the frame for a universe survey, 
the 1989-90 Private School Survey (PSS). The list frame for the 1989-90 PSS was used to select 
the private school list sample for Round 2 of SASS. As in Round 1, the list frame was 
supplemented by an area sample, but the number of sample areas was increased from 75 to 123 
(Gruber, Rohr, and Fondelier, 1993). 

The sample design in Round 2 was modified to improve the estimates for certain domains. For 
instance, in the School Survey, domains of Bureau of Indian Affairs (BIA) schools and Native 
American schools (schools with 25 percent or more Native American students) were added. In 
the Teacher Survey, separate domains were included for Asian and Pacific Islander (API) 
teachers and for American Indian, Aleut and Eskimo (AIAE) teachers. 

A new feature of the Round 2 sample selection process was a procedure to control the amount of 
overlap between the Round 1 and 2 school samples. The proportion of overlap varied by sector 
and by stratum within sector, based on an evaluation of the tradeoff between improved estimates 
of change (favoring more overlap) and expected effects on response rates (favoring less overlap) 
(Kaufman and Huang, 1993, Section 4). Additional design changes are described in chapters 2 
to 6 for each of the SASS surveys. 

The separate School and TDSS questionnaires used for private schools in Round 1 were 
combined to form a single Private School questionnaire in Round 2. Also, a new questionnaire 
was added for BIA schools outside the local public school system. This questionnaire also 
includes items for the School and TDSS. 

Round 3. Round 3 incorporated further changes in content, design, and procedures. The Student 
Records Survey became a new component of SASS. Procedures for obtaining student data from 
school records were tested in the spring of 1993 and used in Round 3 for students in a subsample 
of schools. Round 3 also added the Library Media Center Survey and the Library Media 
Specialist/Librarian Survey, conducted among a subsample of schools in the School Survey. 

The survey questionnaires were modified in various ways. Some questions that required 
considerable effort but yielded little usable information were dropped; others were reformatted 
for easier completion by respondents. Instructions for skipping items not relevant to all 
respondents were clarified. New items were added to provide information needed to monitor the 
National Education Goals for the year 2000, including data on topics such as school safety and 
drug use by students. 



Round 4. In Round 4, charter schools have been included as a new domain, with' separate data to 
be provided for this group of schools, their principals, and their teachers. The Round 4 SASS, 
covering the 1999-00 school year, also shifts the survey emphasis from teacher supply and 
demand issues to the measurement of teacher and school capacity, both objectives of recent 
school reform agenda (see Mullens and Kasprzyk, 1997; NCES, 1999). Since there is evidence to 
suggest that schools and school districts may reconcile imbalances in teacher supply and demand 
by adjusting the level of teacher quality accepted (see Bobbitt, 1995; Broughman and 
Rollefson, 1995), issues of teacher quality are of increased concern. The Round 4 SASS 
highlights issues pertaining to teacher qualifications and career paths, and places more emphasis 
on teacher professional development. 

o 

Periodicity. Rounds 1 to 3 of SASS were administered at 3-year intervals, in 1987-88, 1990-91, 
and 1993-94. The interval between Round 4, administered in 1999-00, and Round 3 was 
extended to 6 years, because of both budget and redesign considerations. SASS is currently 
planned on a 4- year cycle; Round 5 will take place in 2003-04. 

Sample design. The sample schools in the SASS School Survey are the core link to other SASS 
samples. Figure 1-1 shows the Round 3 SASS samples and the links between them. The sample 
selection involves the following steps: 

(1) A sample of schools is selected for the School Survey. The sample is designed to provide 
separate estimates for public, private, BIA, and Native American schools. 

(2) The same sample of schools selected for the School Survey is used for the School Principal 
Survey. 

(3) Each LEA that administers one or more of the public schools selected for the School Survey 
becomes part of the sample for the TDSS. Each round also included a supplemental sample 
of LEAs that did not administer schools. For the private sector and BIA schools which are 
not part of an LEA, the data for the TDSS are collected from the schools sampled for the 
School Survey as part of the questionnaire for that survey. 

(4) For each sample school, a list of teachers is obtained and a sample of teachers is selected for 
the Teacher Survey. 

(5) The Teacher Followup Survey takes place in the school year following the basic SASS 
surveys. Its sample is selected from the respondents to the Teacher Survey and includes a 
subsample of those who continue teaching at the same or another school and all those who 
are no longer teaching in an elementary or secondary school. 

(6) A subsample of schools included in the School Survey is selected for the Library Media 
Center and Library Media Specialist/Librarian Surveys. 

(7) Another subsample of schools (selected with minimum overlap with the schools selected for 
the Library Media Center and Library Media Specialist/Librarian Surveys) is selected for the 
Student Records Survey. Within these schools, the sample teachers for the Teacher Survey 
are subsampled, and students are selected from the sample teachers’ class rosters. 
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Figure 1-1. — ^The Round 3 (1993-94) SASS samples 




*For private schools and certain BIA schools, the Teacher E>emand and Shortage Survey is part of the School Survey. 
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Table 1-1 shows the sample sizes for the S ASS surveys in Rounds 1 throu^ 3 and the 
approximate sample sizes in Round 4. The school counts in Round 1 reflect a somewhat 
different definition of schools from that used in subsequent rounds; the difference, however, is 
small (see section 2.4). Round 4 also includes samples of about 1,100 charter schools and their 
principals, and 5,000 teachers who work in them. 

Table 1-1. — Sample size of each survey in SASS, by sector: Rounds 1-4 







Number sampled 




SASS surveys and sectors 


Round 1* 
(1987-88) 


Round 2 
(1990-91) 


Round 3 
(1993-94) 


Round 4^ 
(1999-00) 


Public 

School 


9,317 


9,587 


9,956 


9,800 


Principal 


9,317 


9,587 


9,956 


9,800 


LEA 


5,594 


5,424 


5,459 


5,500 


Teacher 


56,242 


56,051 


.56,736 


57,000 


Library Media Center 


4 


. ■ . . 4 


4,994 


9,800 


Library Media Specialist/Librarian 


4 


4 


4,994 


4 


Student records 


4 . 


■ ■ 4 


5,095 


4 


Private 

School 


3,513 


3,270. 


3,360 


3,400 


Principal 


3,513 


3,270 


3,360 


3,400 


Teacher 


11,529 


9,166 


11,548 


12,000 


Library Media Center 


4 


4 


2,500 


3,400 


Library Media Specialist/Librarian 


4 


4 


2,500 


4 


Student records 


4 


4 


1,236 


■ 4 


BIA 

School 


■ 4 


101 


176 


200 


LEA 


4 


• 101 


176 


200 


Teacher 


4 


— 


— . , 


1,000 


Library Media Center 


+ 

X 


4 


122 


200 


Library Media Specialist/Librarian 


4 


4 


122 


4 


Student records 


4 


4 


602 


4 



^Not applicable. (The survey was not administered in that round.) 

“T^ot available. 

*The Round 1 public school sample includes the BIA schools listed separately in Rounds 2 to 4. Round 1 also used a slightly different definition 
of schools; the difference, however, was small (see section 2.4). 

^The sample sizes for Round 4 are approximate. Round 4 also includes samples of about 1,100 charter schools and principals and about 5,000 
teachers who work in them. These samples are not included in this table. 

SOURCES: NCES (1991a,b,c,d); Gruber, Rohr, and Fondelier (1993,1996); NCES (1999). 



Survey Operations. The main survey operations, including sample selection, data collection and 
data processing, are carried out by the U.S. Census Bureau under an interagency agreement, 
according to specifications. provided by NCES. Questionnaires are mailed to sample schools, 
addressed to the principal or directly to sample teachers. For the Teacher Survey in Round 1 



only, teacher questionnaires were mailed to school coordinators who distributed them to the 
sample teachers in their schools. Completed questionnaires are returned by mail to the Census 
Bureau's clerical processing office in Jeffersonville, Indiana. 

The data collection processes for SASS typically involve an initial mailing of the questionnaires, 
a second mailing to nonrespondents, and followup telephone interviews to mail nonrespondents. 
Round 3 introduced the use of a reminder postcard sent about one week after the initial mailing 
to all sample units. Prior to Round 3, nonrespondents to the second mailing were contacted by 
telephone by U.S. Census Bureau interviewers — known as field representatives — working from 
their homes or from Census regional offices, using paper-and-pencil questionnaires. In Rounds 
3 and 4, nonresponse followup has mostly been accomplished through centralized computer- 
assisted telephone interviewing (CATI) by U.S. Census Bureau field representatives. LEA 
nonrespondents were followed up by regional field office staff using paper-and-pencil 
questionnaires in Round 3; they are designated for CATI followup in Round 4. 

Table 1-2 summarizes the data collection schedules for Rounds 1 to 4. In an attempt to increase 
response rates and to allow time for completing all components of the data collection activities, 
the initial mailouts of questionnaires have moved progressively earlier over the four rounds, in 
order to reach the different respondents, whether principals, teachers, or school officials, earlier 
in the school year. Thus, whereas in Round 1 the data collection effort ran from November to 
June, by Round 4 it began in August and is scheduled to end in June. 

Use of SASS data. SASS produces a range of products, including publications and data files 
stored on tapes or CD-ROMs. Disseminating SASS data to the greatest number of users and in a 
form that is appropriate, usable, and accessible is an ongoing concern in SASS. The technical 
report SASS Schools and Staffing Survey (NCES, 1999) provides an overview of SASS and a 
summary of SASS publications. Two separate studies address the issue of customer needs and 
satisfaction. Tracking Secondary Use of the Schools and Staffing Survey Data: Preliminary 
Results (Wiley and Reynolds, 1999) identifies individuals and institutions using SASS data for 
research purposes. What Users Say About Schools and Staffing Survey Publications (Rouk, 
Weiner, and Riley, 1999) reviewed the purposes of SASS publications, based on a survey of 
users and comments collected from a focus group discussion on how SASS data are 
communicated in print. The results of these studies will be used to improve the dissemination of 
SASS data with the aim of expanding their use. 

1.3 SOURCES OF INFORMATION ABOUT SASS 

Sources of information about sampling and nonsampling errors. Errors in surveys are of two 
kinds: sampling and nonsampling errors. Sampling errors are the result of basing survey 
estimates on a sample rather than all units in the population of interest. Nonsampling errors can 
occur at any stage of a survey, including sample selection, data collection, data processing, and 
estimation. This report covers both sources of error; however, the primary emphasis is on the 
nonsampling error. 

Sampling errors. SASS data users requiring detailed guidance on how to take sampling errors 
into account in their analyses should refer to the SASS Data File User's Manual provided with 
the data files for each round (NCES, 1991a, b, c, d; Gruber, Rohr, and Fondelier, 1993 and 
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Table 1-2. — SASS data collection schedules: Rounds 1-4 



Form/questionnaire 


Initial 

mailout 


Begin 

followup 


Complete 

collection 


Round 1 (1987-88) 
Teacher listing form 


November 1987 


December 1987 


January 1988 


School district (public schools) 


January 1988 


April 1988 


June 1988 


School principal 


January 1988 


April 1988 


June 1988 


School 


January 1988 


April 1988 


June 1988 


Teacher 


January 1988 


April 1988 


June 1988 


Teacher followup 


■ March 1989 


April 1989 


July 1989 


Round 2 (1990-91) 


Teacher listing form 


October 1990 


November 1990 


December 1990 


School district (public schools) 


December 1990 


March 1991 


June 1991 


School principal 


January 1991 


March 1991 


June 1991 


School 


January 1991 


March 1991 


June 1991 


Teacher 


January 1991 


March 1991 


June 1991 


Teacher followup 


January/February 1992 


March 1992 


May 1992 


Round 3 (1993-94) 
Teacher listing form 


September 1993 


November 1993 


February 1994 


School district (public schools) 


October 1993 


January 1994 


June 1994 


School principal 


October 1993 


January 1994 


June 1994 


School 


December 1993 


March 1994 


June 1994 


Teacher 


December 1993 


March 1994 


June 1994 


Library/librarian 


October 1993 


March 1994 


June 1994 


Student records 


March 1994 


March 1994* 


June 1994 


Teacher followup 


January 1995 


February 1995 


May 1995 


Round 4 (1999-00) 


Teacher listing form 


August 1999 


October 1999 


Februaryl999 


School district (public schools) 


September 1999 


January 2000 


May 2000 


Principal 


September 1999 


November 1999 


April 2000 


School 


September 1999 


January 2000 


April 2000 


Teacher 


November 1999 


January 2000 


June 2000 


School library media center 


September 1999 


November 1999 


April 2000 



♦For those schools that would not select student samples by phone, a personal visit to the school was arranged, beginning in March, to select the 
student sample and to complete the student records questi<Minaire. Followup for other schools that had not returned completed student records 
questionnaires began in April. 

SOURCES: NCES (1991 a,b,c,d); Gruber, Rohr, and Fbndelier ( 1 993, 1996); NCES ( 1 999). . 

1996). All publications based on SASS data include infonnation about sampling errors of SASS 
estimates. Each publication in the Education Data Tabulations (ED Tabs) series (e.g., Bobbitt, 
Gruber, and Leich, 1993; Bobbitt, Broughman, and Gruber, 1995) includes separate tables with 
sampling errors for selected estimates included in the publication. The publications that 
summarize results of SASS for each round (Choy, Mednch, Henke, and Bobbitt, 1992; Choy, 
Henke, Alt, Medrich, and Bobbitt, 1993; Henke, Choy, Geis, and Broughman, 1996) include 
tables showing the estimated standard error for each estimate discussed in the text of the report 
and a table showing standard errors for selected public school estimates at the state level. 
Generalized variance functions, which provide approximations of sampling errors for all 
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estimates, based on their size, have been developed for the School, Principal, TDSS, and Teacher 
Surveys for Rounds 1 and 2 (Salvucci and Holt, 1992; Salvucci, Galfond, and Kaufman, 1993; 
Salvucci, Holt, Moonesinghe, and Kaufman, 1994; Salvucci, Weng, and Kaufman, 1995). 

Nonsampling errors. Information about nonsampling errors comes from several sources: 

■ Operational or performance data, including unit and item response rates, results of 
supervisory reviews of interviewers' work, results of reinterviews, and pre-edit and edit 
failure rates. 

B Findings from pretests, in-depth group and individual interviews, and methodological 
experiments. 

B Micro-evaluation studies, in which the accuracy of a sample of individual responses is 
evaluated by various means, such as intensive reinterviews or comparison with existing 
records. 

B Macro-evaluation studies, in which the differences between survey estimates and comparable 
estimates from other sources are analyzed. Such studies may involve data from two or more 
SASS surveys or they may compare SASS data with those from other NCES surveys or from 
surveys conducted by other organizations, such as the U.S. Census Bureau. 

This report draws on all of these sources. It is based almost entirely on. existing documentation 
with minor exceptions; no new tabulations or analyses were undertaken to provide new material. 

Additional sources of information. The references cited in this report include several kinds of 
NCES publications. The Data File User's Manuals (sometimes referred to as codebooks) are the 
most comprehensive source of information about the data available for analyses. 

B Round 1 — ^In Round 1, five separate 1987-88 Data File User's Manuals (NCES, 1991 
a,b,c,d; Faupel, Bobbitt, and Friedrichs, 1992) are available, one for each of the basic SASS 
surveys — School, School Principal, TDSS, and Teacher — and one for the Teacher Followup 
Survey. The manual for each of the four basic surveys contains a common section describing 
the survey design and procedures for these surveys, followed by documentation of the data 
files and copies of the questionnaires for the specified survey. The manual for the Round 1 
Teacher Followup Survey is specific to that survey. 

B Round 2 — For Round 2, the 1990-91 Data File User's Manual covers all four of the basic 
surveys (Gruber, Rohr, and Fondelier, 1993). Volume 1 describes the design and procedures 
for the surveys. Volume II provides documentation of the restricted-use survey data files, and 
Volume in provides documentation of the public-use data files. A separate 1990-91 Data 
File User's Manual also has been released for the Round 2 Teacher Followup Survey 
(Whitener, Rohr, Bynum, Kaufman, and King, 1994). 

B Round 3 — ^The Round 3 1993-94 Data File User's Manuals for the basic surveys (Gruber, 
Rohr, and Fondelier, 1996) and the Teacher Followup Survey (Whitener, Gruber, Rohr, and 
Fondelier, 1998) follow the same structure as the Round 2 manuals but provide more detailed 
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information about the edit and imputation procedures applied to each data item in the 
questionnaires. 

Another useful source of information about the SASS surveys is the Sample Design and 
Estimation Report for each round. Kaufman (1991) gives a detailed description of the sample 
design and estimation for the four basic surveys in Round 1. Comparable reports are available 
for Round 2 (Kaufman and Huang, 1993), and Round 3 (Abramson et al., 1996). 

A third source consists of the papers presented at the annual meetings of the American Statistical 
Association (ASA) on various aspects of SASS methodology and the quality of SASS data. 
These papers are generally published in the Proceedings of the Section on Survey Research 
Methods. NCES also publishes a working paper almost every year that compiles selected ASA 
papers concerned with SASS (NCES, 1994, 1995, 1996, 1997). 

Users can gain access to SASS publications, data, and questionnaires from the SASS Home 
Page, http://nces.ed.gov/surveys/sass. They can also communicate with SASS staff through 
email to sassdata@ed.gov; this address gives a list of addresses of contact persons who can 
assist users with SASS questions. 

This report relies on published sources whenever possible, but much of the information comes 
from unpublished memoranda and reports. Readers who would like to obtain copies of these 
items or who have questions about SASS findings and methodology should write to: 

SASS Quality Profile 

National Center for Education Statistics 

1990 K Street, N.W. 

Washington, D.C. 20006 
sassdata@ed.gov 
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2. THE SCHOOL SURVEY 



2.1 INTRODUCTION 

The School Survey is the core survey in the Schools and Staffing Surveys (SASS), providing 
basic data on elementary and secondary schools in the United States and serving as a sampling 
frame for the other surveys in SASS. The School Survey has provided school data for 1987-88 
in Round 1, 1990-91 in Round 2, and 1993-94 in Round 3 and data are currently being collected 
for 1999-00 in Round 4. The sample for Rounds 1 to 3 consisted of about 13,000 schools, of 
which almost three-quarters were public schools and the remaining quarter were private schools. 
Close to one-third of the schools provided data for two consecutive rounds. The sample size 
meets the key requirements of producing state estimates for public schools, estimates by 
association or affiliation group for private schools, and beginning in Round 2, estimates for 
schools listed on the Bureau of Indian Affairs' (BIA) Program Education Directory. Round 4 
added a sample of over 1,100 charter schools to provide national estimates for such schools. 
Schools receive questionnaires by mail, and with telephone followup of mail nonrespondents, the 
response rates for the School Surveys have been high: over 95 percent for BIA schools, 92 
percent for public schools, and almost 85 percent for private schools. 

The survey questionnaires in Rounds 2 and 3 consisted of three separate forms: public schools 
receive form SASS-3A, private schools receive form SASS-3B, and BIA schools outside the 
local public school system receive form SASS-3C. A new form has been added for charter 
schools in Round 4. The core components of these questionnaires are quite similar and include 
basic school characteristics, staffing patterns, and school policies. The forms for private schools 
and BIA schools also contain items pertinent for these schools and items that appear on the 
Teacher Demand and Shortage Survey (TDSS) for local education agencies (LEAs) of public 
schools. Section 2.5 reviews the contents of the questionnaires for each round. 

The rest of this chapter summarizes the quality issues of the School Survey in seven sections: 
definitions and types of schools in the survey (2.2); sample design choices (2.3); the school 
samples — frame, selection, sample size, and coverage issues (2.4); questionnaire content (2.5); 
data collection, measurement error studies, and nonresponse (2.6); data processing and 
estimation (2.7); and evaluations of school estimates (2.8). 

2.2 DEFINITIONS AND TYPES OF SCHOOLS 

The target population for the School Survey is all elementary and secondary schools in the 
United States in operation during the reference school year (1987-88, 1990-91, 1993-94, and 
1999-00. Schools with no students in any of grades 1 to 12 are considered out of scope and 
excluded. Thus, schools offering only kindergarten, prekindergarten, postsecondary, and adult 
education programs are excluded. 

Elementary schools are schools with a grade below 7 and no grade above 8. Secondary schools 
are schools with any span of grades beginning with grade 7 or above and ending with or below 
grade 12. Combined schools are schools with any span of grades that includes a grade lower 
than 7 and a grade higher than 8. Nonregular schools, such as special education, vocational. 
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technical, adult education (if part of an in-scope school), and alternative or continuation grade 
schools, are classified as combined schools. 

A public school is defined as an institution that provides educational services, has one or more 
teachers to give instruction, is located in one or more buildings, receives public funds as primary 
support, and is operated by an education agency. An additional requirement beginning at Round 
2 is that the institution has an assigned administrator. This definition of public school includes 
schools in juvenile detention centers, schools associated with publicly operated hospitals, schools 
located on military bases and operated by the U.S. Department of Defense (DOD), and schools 
affiliated with the BIA and operated by LEAs. BIA schools operated outside the local public 
school system are excluded. 

A BIA school is defined as an educational or residential center funded by and operated by, or 
under contract with, the BIA that offers services to American Indian students. The school can 
occupy one or more buildings and may be a day school, a boarding school, previously a private 
school, a cooperative school, or a contract school. Schools operated outside the local public 
school system by Indian tribes or by Indian tribes under contract with the BIA also are classified 
as BIA schools. 

A Native American school is a public school with 25 percent or more American Indian or 
Alaskan Native students. 

A charter school is a public school that, in accordance with an enabling state statue, has been 
granted a charter, exempting it from selected state or local rules and regulations. A charter 
school may be a newly created school or it may have converted from a pre-existing public or 
private school. 

A private school is a school not in the public system that provides instruction for any of grades 1 
to 12 where the instruction is not given exclusively in a private home. A private school is out of 
scope if it has no students in any of grades 1 to 12, if it operates in a private home used as a 
family residence, if it has fewer than 10 students, or if it has only one teacher. 

Private schools in the United States are diverse in orientation and affiliation. Many private 
schools are affiliated with a religious body (e.g.. Catholic, Evangelical Lutheran Church in 
America, and Seventh Day Adventist). Other private schools are members of an association of 
schools (e.g.. National Association of Independent Schools and National Association of Private 
Schools for Exceptional Children). Some schools with religious affiliations are also members of 
associations and some religious schools do not have a formal association with any organized 
religious group or religious association. The SASS School Survey treats some of the association 
groups as subdomains for which separate estimates of adequate precision are required (the 
association groups are shown in table 2-5). 

2.3 SAMPLE DESIGN CHOICES 

The sample design for the School Survey is a compromise design that meets the multiple 
objectives for SASS and takes into consideration the response burden for schools. This section 
discusses the four basic goals of the survey and how each of these goals guided the choice of 
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sample design. The four goals discussed are the following: providing reliable domain estimates; 
balancing the requirements of the School Survey and other. S ASS surveys; controlling sample 
overlap between SASS and other National Center for Education Statistics (NCES) surveys of 
elementary and secondary schools; and controlling sample overlap between rounds of SASS 
surveys. 

Providing domain estimates. The main design objective of the School Survey is to provide 
estimates of school characteristics by key analytical domains. The following are domains for 
school estimation: 

■ The nation as a whole; 

■ The public and private sectors separately; 

■ Elementary, secondary, and combined schools,. separately for the public and private sectors; 

■ BIA and Native American schools separately (since Round 2); 

■ Elementary and secondary public schools, separately by state (since Round 2); and 

■ Private schools by association group, region, and school level (elementary, secondary, and 
combined). 

The optimum allocation for each of the key domains of estimation is different. For national 
estimates, the number of sample schools should be made proportional to the total number of 
schools within each stratum. However, for state comparisons, the optimal allocation is one in 
which each state gets an equal sample allocation. Using the optimum allocation for state 
comparisons would provide less efficient national estimates (e.g., California would get the same 
sample size as Delaware even though California contributes more to the national variance than 
Delaware). Likewise, the optimum allocation for sector is different from the optimum allocation 
for level. In other words, optimizing the allocation for one goal results in less efficient estimates 
for another goal. For this reason. Round 1 of the School Survey used a compromise allocation 
that took into account the loss of efficiency relative to the optimum allocation for each of the 
four domains (national, state, level, and sector) and for both school and teacher estimates (see 
Kaufman, 1991). 

The sample allocations for subsequent rounds modified it somewhat to improve the precision of 
elementary and secondary school estimates by state for the public school sector. For the private 
school sector, the design was modified to support estimation by private school association group, 
census region, and school level. 

Section 2.4 describes in more detail the sample allocation and design changes for the public and 
the private school samples by round. The overall design to support estimation by domain 
involves stratifying schools by sector and by domain within each sector. The allocation of 
sample schools among the strata is designed to provide estimates of acceptable precision for each 
of the analytical domains. The general approach is to select a specified minimum number of 
schools (about 80 for public schools and 100 for private schools) in each of the domains and to 
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allocate the remaining sample of schools in a way that optimizes the precision of estimates 
aggregated over domains, such as national estimates for public and private schools. 

Balancing the requirements of the samples in SASS. As the core survey in SASS, the School 
Survey serves as a sampling frame for the other SASS surveys. The links between the school 
sample and the samples for the other SASS surveys are displayed in figure 1-1 in chapter 1. 
These links led to the need for some tradeoffs in the sample design for the School Survey. The 
tradeoffs involving the LEA sample associated with public schools and the sample of teachers in 
all schools are summarized below. 

Sampling schools and LEAs. For the public school sector, two design alternatives were 
considered in Round 1 for linking the school and the LEA samples. One alternative starts with 
the selection of a sample of LEAs and then selects a sample of schools in those LEAs. The other 
alternative selects a sample of schools first and then identifies the LEA sample as the LEAs in 
which the sampled schools are located. 

The tradeoff between the two designs is that the LEA-first design improves the reliability of the 
LEA estimates but reduces the reliability of both school and teacher estimates relative to the 
school-first design. Wright (1988) compared these two design alternatives in a simulation study. 
The results showed that the LEA-first design produced less reliable school and teacher estimates 
than the school-first design; however, the school-first design produced only slightly less reliable 
LEA estimates than the LEA-first design. Based on these results, SASS has been using the 
school-first design since Round 1. 

Sampling schools and teachers. The design issue involving the relationship of the school and the 
teacher samples concerns the school selection probabilities: 

■ The greatest precision for school estimates would be achieved by selecting schools with 
equal probability within each stratum. However, this design is inefficient for the teacher 
sample. If schools are selected with equal probability, within each stratum teachers need to 
be selected within sampled schools at a constant rate in order to give them overall equal 
probabilities within the stratum. This design leads to a large variation in the number of 
teachers selected per school and undue response burden for some schools. The undue burden 
could be avoided by restricting the teacher sample size per school, but that destroys the equal 
probability feature of the teacher sample, and hence leads to the need for weighting 
adjustments in the analysis. 

■ For the teacher sample, an optimum design would select schools with probabilities 
proportionate to estimated size (i.e., proportionate to the number of teachers at the schools on 
the frame), and then would select teachers within schools at a rate that would make the 
overall selection probability for teachers constant within strata. However, this design would 
result in schools with many teachers having considerably higher probabilities of selection 
than schools with few teachers. The large variation in the selection probabilities of schools 
would result in a major loss of precision in the school estimates. 

In place of either of these approaches, SASS adopted a compromise solution that balances the 
requirements of the two samples, namely, selection of schools with probability proportionate to 
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the square root of the number of teachers. Teachers within schools are then sampled at a rate 
that makes the overall selection probability for teachers approximately constant within strata, 
subject to the constraints of sampling at least one and no more than 20 teachers per school. The 
square root of the number of teachers is the geometric mean between the two alternative 
measures of size discussed above — the number of teachers as is optimal for the teacher sample 
and unity as is optimal for the school sample. The use of this compromise allocation avoids the 
substantial losses in precision for one of the surveys by using the measure of size that is optimal 
for the other survey. With the compromise measure of size, both surveys should be subject to 
only modest losses in precision compared with their optimum designs. The square root 
compromise measure of size is also used in some other surveys with differing units of analysis, 
such as some surveys in the NCES Fast Response Survey System. 

Controlling sample overlap between SASS and other NCES school surveys. NCES periodically 
conducts a number of surveys of elementary and secondary schools, and the data collection 
periods of its school surveys sometimes overlap. Accordingly, an initial design concern in 
Round 1 was to minimize any sample overlap between the School Survey and any other 
concurrent NCES school surveys in order to avoid the burden on schools of having to respond to 
more than one survey at the same time. 

During the data collection period for Round 1, NCES was also collecting data for two other 
surveys of elementary and secondary schools: the National Assessment of Educational Progress 
(NAEP) and the National Educational Longitudinal Study (NELS) of 1988. All three surveys 
used the Quality Education Data (QED) file as the basic sampling frame, but they had different 
selection probabilities and frame specifications. To minimize the response burden on individual 
schools, the samples for these three surveys were selected in a manner that minimized the 
overlap in sampled schools. A sequential procedure was used. First the NAEP sample was 
selected, then the NELS sample was selected using a procedure that avoided overlap with the 
NAEP sample while retaining the NELS selection probabilities (see Spencer et al., 1990), and 
finally the SASS sample was selected using a procedure that minimized the overlap with NAEP 
and NELS while retaining the SASS selection probabilities (see Kaufman, 1993, appendix 1). 
The procedure limited the overlap in the SASS sample to four schools in common with the 
NAEP sample. The SASS sample had no schools in common with the NELS sample. 

A comparable procedure was not required in Round 2 because no other NCES surveys were in 
the field at the time. Round 3 of SASS coincided with the 1993-94 NAEP data collection, but no 
control over sample overlap was employed for this round. Kaufman (1999) found no evidence of 
a negative effect on school response rates to Round 3 of SASS due to sample overlap with the 
1993-94 NAEP. 

Round 4 of SASS is being conducted in the same school year as the 1999-00 NAEP. The NAEP 
sample was selected first, and then the SASS sample was selected using a procedure to minimize 
overlap. Since there is interest in the joint analysis of the school data collected in SASS and the 
student achievement data collected in NAEP, a special study is also planned to collect both sets 
of data from a supplemental sample of about 800 public schools (400 schools with grade 4 and 
400 schools with grade 8). The benefits of a SASS-NAEP linkage are discussed by Skaggs and 
Kaufman (2000). 
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Controlling sample overlap between rounds. Some schools sampled for one round of SASS may 
also be sampled for the following round. The decision about how much the samples for two 
successive rounds should overlap involves a tradeoff between the anticipated favorable and 
unfavorable effects of overlap. To provide estimates of change between rounds with maximum 
precision, the overlap should be as great as possible. However, it was thought that response rates 
for schools selected for a second time might be lower than for those selected for the first time. 
The overlap issue is further complicated by the fact that overlap in the sample of public schools 
guarantees overlap in the sample of LE As associated with them.’ A Round 2 pretest conducted 
early in 1990 estimated that overlap would reduce the response rate by 5 percent for schools and 
1 1 percent for LEAs. 

The decision at Round 2 was to set the sample overlap for public schools at 30 percent, with a 
resultant LEA overlap rate of about 58 percent. Private schools had substantially lower response 
rates than public schools in Round 1 (see table 2-11), which raised concerns about the extent of 
overlap. For schools in association groups not sampled with certainty, the decision was to 
control, where feasible, the private school overlap of the list sample at 30 percent for school 
association groups with high response rates (close to 85 percent or more), to minimize overlap 
for the association groups with low response rates (below 80 percent), and to aim at a level less 
than 30 percent, but more than the minimum, for the association groups with response rates 
between 80 and 85 percent (see Kaufman and Huang, 1993, part 4). Another component of the 
private school sample is a saniple of unlisted schools found in a sample of areas (primary 
sampling units or PSUs). For this component, SASS uses the area sample originally selected for 
the Private School Survey (PSS); the PSS control for PSU overlap is discussed in section 2.4.2. 
The SASS area sample schools were selected independently in the PSUs selected for both rounds 
with no attempt to control overlap between rounds. 

The decision at Round 3 was to maintain the same sample overlap rates as Round 2 for both 
public and private schools. The target sample overlap for public schools was again set at 30 
percent. For private schools, the control of overlap for each association group used the same 
approach as that used in Round 2. The procedures used to control overlap are discussed in 
Abramson et al. (1996, appendix 2). Round 4 does not control for sample overlap with Round 3 
because of the longer interval between rounds. 

2.4 SELECTING THE SCHOOL SAMPLE 

The samples of public and private schools for the School Survey are selected using different 
frames and procedures. This section discusses the public school samples in Rounds 1 to 3, for 
example, frame construction, sample selection, and sample sizes (2.4.1); the private school 
samples — dual frames, sample selection, and sample sizes (2.4.2); and coverage and other frame 
issues (2.4.3). The estimation of sampling errors is discussed in section 2.7.5. 



* The LEA overlap rate for Rounds 1 and 2 was expected to be about 47 percent if schools in the two successive rounds were selected as 
independent samples. 
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2.4.1 Public school samples 

The sampling frame for the Round 1 public school sample was the 1985-86 QED, an annual 
database prepared by Quality Education Data Incorporated, a private research and database 
company in Denver, Colorado. The primary frame for Rounds 2 and 3 was the CCD, an NCES 
database compiled from administrative record data reported annually by state education agencies 
(SEAs). Round 2 used the 1988-89 CCD school file, and Round 3 used the 1991-92 CCD school 
file. Both the QED and the CCD files are universe files that are intended to include all 
elementary and secondary schools in the United States. McMillen, Kasprzyk, and Planchon 
(1994) discuss the sampling frames managed by NCES and efforts to assure quality and integrate 
the sampling frames for major NCES surveys. 

QED and CCD schools. The CCD and the QED define schools slightly differently. While 
schools on the QED are listed as physical locations, those on the CCD are shown as 
administrative units. For example, an elementary school and high school at the same physical 
location but with two different principals would be counted as one school on the QED list but as 
two schools on the CCD list. In contrast, a consolidated school district might have a high school 
that gathers in two buildings, at two physical locations, but is administered as one high school. 
This case would be listed as two schools on the QED but as one school on the CCD. 

Frame construction. The construction of the public school sampling frame at each round 
included steps to remove duplicate schools, schools operated by the DOD outside the United 
States, and schools that offered only kindergarten and prekindergarten or adult education. As an 
example, the 1991-92 CCD school file used as the primary list frame for Round 3 contained a 
total of 86,287 public schools. Frame cleaning removed 3,451 schools (about 4 percent of the 
schools on the CCD file), leaving 82,746 schools that were considered to be in-scope for SASS. 
For the preceding rounds, the public school sampling frame contained 80,384 in-scope QED 
schools in Round 1 and 83,165 in-scope CCD schools in Round 2. 

The public school sample for each round was a stratified probability sample. The allocation of 
sample schools among the strata was designed to provide estimates of acceptable precision for 
each of several analytical domains (see section 2.3). The Sample Design and Estimation reports 
(Kaufman, 1991; Kaufman and Huang, 1993; Abramson et al., 1996) provide the details of the 
sampling processes in each round. Briefly, schools in the frame were, first stratified by analytic 
domain. Within each stratum, the schools were next sorted by LEA characteristics (e.g., 
urbanicity, percent minority student enrollment, and ZIP Code) and by school characteristics 
(e.g., highest grade in the school and school enrollment). Following the sort operation, the 
specified number of schools was selected systematically with schools selected with probability 
proportionate to the square root of the number of teachers as reported on the frame. The 
procedure of systematic sampling from the sorted lists was erriployed in order to satisfy the 
requirements of the LEA sample and to gain the benefits of implicit stratification by the sort 
variables. Prior to sampling, all schools with a measure of size that exceeded the sampling 
interval for the stratum were removed from the sampling frame and selected with certainty. For 
the remaining schools, the controls on overlap with the sample from the preceding round were 
built into this part of the selection process (see Kaufman and Huang, 1993, appendix 3, and 
Abramson et al., 1996, appendix 2). 



Beginning with Round 2, the LEAs in Delaware, Nevada, and West Virginia were treated as 
separate strata and at least one school was sampled per LEA thereby ensuring that all LEAs in 
these states were included in the LEA sample. , Prior to Round 2, a simulation study was 
conducted to assess the reliability of SASS LEA estimates for each state. .The study showed that 
the standard errors of LEA estimates for these three states were very high even with a very high 
sampling rate. By including all LEAs in the sample with certainty, there are no sampling errors 
for LEAs estimates in these states (see Kaufman and Huangs 1993, p. 45). 

Sample sizes. Table 2-1 shows the sample sizes of public schools in the School Survey in 
Rounds 1 to 3. Round 1 used the QED frame and the sample allocation designated the selection 
of a sample of 9,331 public schools. The actual number of schools sampled, 9,317 schools, was 
slightly different from the designated sample size because of the procedures included to control 
for sample overlap with other surveys. Rounds 2 and 3 used CCD frames, and BIA schools and 
Native American schools were assigned to separate strata for sampling. To facilitate 
comparisons with Round 1, Round 2 sampled CCD schools grouped according to the QED 
school definition. After sample selection, the CCD definition of a school was applied to the 
selected units, which resulted in a slightly larger sample of schools. As explained earlier, CCD 
defines schools based on administrative units and QED defines schools based on physical 
locations. There can be multiple school administrative units at one physical location. Round 3 
sampled schools as listed on the CCD frame. The actual number of schools sampled differed 
slightly from the sample allocation because of the procedures included to control sample overlap 
with Round 2. 



Table 2-1. — Allocated and actual sample sizes of public schools in the School Survey: 
Rounds 1-3 



Types of public school 




Sample size* 




Round 1 
(1987-88) 


Round 2 
(1990-91) 


Round 3 
(1993-94) 


Total sample allocation ■ 


9,331 


9,688 


9,960 


Public 


9,331 


9,336 


9,333 


BIA 




101 


176 


Native American 


t . 


251 


451 


Combined 


1,430 


1,531 


1,396 


Elementary 


4,994 


4,435 


4,542 


Secondary 


2,907 


3,722 


4,022 


Actual sample selected 


9,317 


9,897 


9,956 



*Not applicable. (Not listed as a separate category.) 

*Round 1 sampled from a QED school frame. Round 2 used a CCD frame; however, schools were grouped by QED school definition for 
sampling, and each CCD school in the sample received a separate questionnaire for data collection. Round 3 sampled schools as listed on the 
CCD frame. 

SOURCES: NCES (1991 a); Gruber, Rohr, and Fondelier ( 1 993, 1 996). 
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The Round 2 sample design allows for the calculation of school, principal, and teacher estimates 
using either the QED or the CCD definition of a school (see Holt and Scanlon, 1995; and 
Kasprzyk et al., 1994). The results of such calculations showed that only a small percent of the 
CCD-defmed schools needed to be adjusted to meet the QED school definition. Only 264 out of 
over 9,300 CCD schools sampled in Round 2 were adjusted to meet the QED definition. These 
schools were mostly in rural or small towns and were concentrated in rix states. North Dakota, 
South Dakota, Iowa, Nebraska, Minnesota, and Texas. 

By school level, the Round 1 public school sample included about 1,400 combined schools, 
5,000 elementary schools, and 3,000 secondary schools. Rounds 2 and 3 allocated a larger 
sample to secondary schools as part of a design change to improve the precision of estimates for 
secondary schools by state. 

Sample size by state. Table 2-2 shows the allocation of the public school sample and the percent 
of schools sampled by state for Rounds 1 to 3. As noted above. Round 1 used a QED frame for 
sampling. Round 2 used a CCD frame with schools on the frame grouped according to QED 
school definition for sampling, and Round 3 used a CCD frame with schools sampled as listed on 
the frame. 

As compared with Round 1, Rounds 2 and 3 allocated larger samples of schools to small states 
and smaller samples to large states in order to improve the precision of the estimates for small 
states. For example, the school sample in Utah rose from 75 schools in Round 1 to 174 schools 
in Round 2; in contrast, the sample size for New York dropped from 480 schools in Round 1 to 
263 schools in Round 2. Kaufman and Huang (1993, appendix 2) investigated the effects of the 
Round 2 allocation on the coefficients of variation^ (CV) of estimates of the numbers of schools 
and teachers from the School Survey. The results showed appreciable gains in the precision of 
estimates for small states and relatively minor losses for large states. 

To control the data collection by state, the fractions of elementary, secondary, and combined 
schools sampled in each state were constrained not to exceed about 40 percent. Subject to this 
constraint, minimum sample sizes of 80 elementary schools, 80 secondary schools, and 10 
combined schools were targeted for each state. With these sample sizes, the CVs of state 
estimates for elementary and secondary schools were expected to be below 15 percent. 

BIA school samples in Rounds 2 and 3. Because of the incomplete coverage of schools operated 
by BIA in Round 1, a separate universe of such schools was established for Rounds 2 and 3, 
drawn from a Program Education Directory maintained by BIA. As a first step, the BIA list was 
matched against the CCD public school universe to identify the BIA schools on the CCD; for 
analytic purposes, such schools were considered as BIA schools. The Round 2 BIA list 
contained 180 schools, of which 152 schools (about 85 percent) were not found on the CCD. 
BIA schools were placed in separate BIA strata by school level and state. Within each stratum, 
these schools were sorted by whether the school was operated by the BIA or by a tribe, and by 
school enrollment, and a sample of 101 schools was selected. BIA schools were selected within 



^ The coefficient of variation (CV) is the ratio of the standard error of an estimate to the mean of the estimate. 



Table 2-2. — Sample allocation and percent of public schools sampled in the School Survey, by 



state; Rounds 1-3 



State 




Allocated sample size^ 




Percent of schools samnled^ 


Round 1 
(1987-88) 


Round 2 
(1990-91) 


Round 3 
(1993-94) 


Round 2 
(1990-91) 


Round 3 
(1993-94) 


Total public schools 


9,331 


9,336 


9,333 


11 


12 


Alabama 


174 


230 


221 


18 


18 


Alaska 


100 


161 


196 


36 


40 


Arizona 


122 


170 


170 


17 


18 


Arkansas 


137 


160 


164 


15 


15 


California 


655 


370 


416 


5 


6 


Colorado 


139 


179 


173 


13 


13 


Connecticut 


128 


170 


170 


18 


18 


Delaware 


72 


72 


72 


44 


42 


EHstrict of Columbia 


72 


74 


72 


40 


40 


Florida 


275 


277 


258 


12 


11 


Georgia 


202 


192 


179 


11 


10 


Hawaii 


74 


92 


94 


40 


40 


Idaho 


103 


158 


166 


28 


29 


Illinois 


357 


241 


283 


6 


7 


Indiana 


190 


200 


184 


10 


10 


Iowa 


163 


174 


169 


11 


11 


Kansas 


146 


161 


161 


11 


11 


Kentucky 


154 


180 


167 


13 


12 


Louisiana 


187 


229 


225 


15 


16 


Maine 


105 


145 


152 


20 


21 


Maryland 


152 


. 178 


171 


15 


14 


Massachusetts 


196 


175 


229 


10 


13 


Michigan 


304 


211 


227 


6 


7 


Minnesota 


177 


189 


171 


12 


11 


Mississippi 


136 


235 


199 


25 


21 


Missouri 


198 


202 


178 


10 


9 


Montana 


106 


163 


161 


21 


20 


Nebraska 


124 


164 


170 


11 


12 


Nevada 


95 


114 


119 


36 


33 


New Hampshire 


91 


118 


120 


27 


30 


New Jersey 


243 


198 


194 


9 


9 


New Mexico 


103 


150 


142 


23 


25 


New York 


480 


263 


313 


7 


8 


North Carolina 


203 


170 


184 


9 


10 


North Dakota 


100 


180 


162 


27 


29 


Ohio 


360 


203 


196 


5 


5 


Oklahoma 


179 


165 


161 


9 


13 


Oregon 


132 


168 


170 


14 


14 


Pennsylvania 


338 


251 


196 


8 


6 


Rhode Island 


80 


108 


106 


36 


35 


South Carolina 


146 


186 


164 


17 


15 


South Dakota 


100 


167 


164 


21 


26 


Tennessee 


94 


205 


189 


13 


13 


Texas 


262 


414 


413 


7 


7 


Utah 


75 


174 


170 


24 


25 


Vermont 


74 


105 


108 


32 


32 


Virginia 


107 


202 


188 


11 


10 


Washington 


88 


176 


197 


10 


11 


West Virginia 


85 


170 


178 


16 


19 


Wisconsin 


98 


166 


170 


8 


9 


Wyoming 


70 


131 


131 


32 


32 



'Round 1 used a QED frame for sampling. Round 2 used a CCD frame with schools grouped by QED school definition. Round 3 used 
a CCD frame and sampled schools as listed on the fraihe. 

^Percent not available for Round 1. 



SOURCES: NCES (1991a); Gruber, Rohr, and Fbndelier (1993, 1996). 



o 

ERIC 



22 



40 



each stratum with probabilities proportionate to the square roots of school enrollments (see 
Kaufman and Huang, 1993). 

In Round 3 the list from BIA's Program Education Directory included 176 schools, 150 of which 
were not found on the CCD list frame. All BIA schools were included in the sample with 
certainty (Abramson et al., 1996). 

Native American school samples in Rounds 2 and 3. For analytic purposes. Native American 
Schools are defined as public schools with 25 percent or more American Indian, Aleut, or 
Eskimo (AIAE) student population. For sampling purposes, the criteria used to assign schools to 
the Native American strata were different for Rounds 2 and 3. For Round 2 the Native American 
strata consisted of schools with 25 percent or more AIAE student populations as recorded on the 
1988-89 CCD. In Alaska, since almost all schools met this criterion, they were sampled under 
the state allocation without separate stratification. For all other states, schools that met the 
Native American sampling criterion for this round were placed into one of four Native American 
strata: Arizona, North Dakota, Oklahoma, and all other states (except Alaska). The stratum 
sample sizes were made proportionate to the numbers of schools in the strata. Within each 
stratum schools were sampled with probability proportionate to the square root of the number of 
teachers as reported on the frame. A total of 251 schools were selected for Round 2. 

Round 3 used a different cutoff for assigning schools into the Native American strata for 
sampling purposes. Instead of 25 percent, schools assigned were those with 19.5 percent or more 
of AIAE student populations as recorded on the 1991-92 CCD. The percent was lowered for 
sampling purpose to broaden the selection criteria, although the analyses continued to be limited 
to schools in which 25 percent or more of the students were AIAE students. Eight Native 
American strata were used: California, Montana, New Mexico, Washington, Arizona, North 
Dakota, Oklahoma, and all other states (again with the exception of Alaska). The sample size 
(excluding Alaska) was 451 schools. 

2.4.2 Private school samples 

Since these is no complete list of private schools, the private school sample for the School 
Survey is selected using a dual frame approach. The approach involves constructing and 
sampling from a private school list frame and supplementing that sample with an area sample of 
schools omitted from the list. This section briefly summarizes the construction of the list frame, 
the area frame, and the selection of the private school samples from these frames in Rounds 1 
to 3. 

List frame. The private school list frame for Round 1 (1987-88) was created using the 1986 
QED file supplemented with lists of schools from 17 national private school association groups. 
The QED list is a list of private schools compiled annually from handbooks, directories, and 
other materials that list private schools. Lists of schools from the association groups were 
matched with the QED list, and Schools not found on the QED list were added to the frame. 
After removing duplicates and out-of-scope schools (e.g., schools that teach only prekindergarten 
and kindergarten students), the SASS Round 1 private school list frame consisted of 22,600 
schools from the QED file and 1,586 schools added from association lists. 
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Round 2 (1990-91) used the list frame developed for the 1989 PSS that was based on the 1988- 
89 QED private school list supplemented by schools not included in that source but found on 
lists supplied by 20 private school associations in the spring of 1990. The Round 2 list frame 
contained 20,600 schools. Round 3 (1993-94) used the 1991-92 PSS list frame updated with 
association lists. The Round 3 list frame contained 24,767 schools. The increase in the number 
of schools on the list frame between Rounds 2 and 3 is accounted for by improvements in the 
PSS list frame. 

The PSS is an NCES survey that includes all private schools on a list frame, supplemented by an 
area sample of unlisted private schools. Since the 1989-90 school year, this survey has been 
administered every 2 years, in 1991-92, 1993-94, 1995-96, 1997-98, and 1999-00 (see 
Broughman, Gerald, Bynum, and Stoner, 1994; Broughman, 1996; and Broughman and 
Colaciello, 1998 and 1999). The PSS target population is all private schools in the United States 
that meet the criteria of a private school described in section 2.2. The list frame is constructed 
from a list of private schools initially developed for the 1989-90 PSS and updated at each 
subsequent administration by matching it with lists provided by nationwide private school 
associations, state departments of education, and other national guides and sources that list 
private schools. The use of state lists of private schools since the 1991-92 PSS has improved the 
coverage of the PSS list frame appreciably (see below). 

The area frame component of the private school sample is based on a listing of private schools 
compiled in a sample of geographical areas for the PSS. The country is divided into a set of 
primary sampling units (PSUs) that are composed of single counties, combinations of contiguous 
counties, or independent cities. A sample of these PSUs is selected for the PSS, and a listing 
operation is conducted in the sampled PSUs. The resultant list is matched with the list frame, 
and all private schools not found on the list frame are retained for the area frame component of 
the private school survey. SASS uses the PSS area frame for its sampling; it does not conduct its 
own independent listing operation. Round 1 was an exception, because of the age of the 1983 
PSS, SASS did not use the PSS area frame schools. Instead an independent field operation was 
conducted in order to update the area frame schools. 

Table 2-3 shows the PSS year for which the area frame was developed, the number of PSUs 
sampled, and the SASS round in which that area frame was used. The Round 1 SASS used the 
same PSU area sample as the 1983 PSS. The area sample for the 1983 PSS consisted of 75 PSUs 
selected out of 2,497 PSUs, each with a population of at least 10,000 people according to the 
1980 Population Census. Eight PSUs with a population greater than 1,700,000 were included 
with certainty. The remaining noncertainty PSUs were selected by systematic sampling with 
probability proportionate to the square root of the 1980 PSU population from 16 strata defined 
by having above or below median student enrollment by census region by metro/non-metro 
status classes. At least three PSUs were selected from each stratum. 

The source of the Round 2 area frame was the 1989-90 PSS. The area sample for the 1989-90 
PSS consisted of 123 PSUs. To attain approximately a 50 percent overlap with the 1983 PSS, 
the sample included the 8 certainty PSUs, 52 overlap PSUs sampled with equal probability from 
the 67 noncertainty PSUs from the 1983 PSS, and 64 PSUs selected independently. The 
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Table 2-3.— Allocation of the area samples for the School Survey: Rounds 1-3 







Area (PSU) sample 


sizes 




SASS rounds 


PSS 


Total 

PSUs 


Certainty 

PSUs 


Noncertainty 

PSUs, 

overlap 

sample 


Noncertainty 

PSUs, 

independent 

sample 


Round 1 
(1987-88) 


1983 


75 


8 


0 


67 


Round 2 
(1990-91) 


1989-90 


123* 


.8 


52 


64 


Round 3 
(1993-94) 


1991-92 


123* 


8 


58 


58 



*One PSU was selected in both the overlap and the independent components. An adjustment was made in the sampling weight to reflect the 
duplicate selection. 

SOURCES: NCES (1991a); Gruber» Rohr» and FondeUer (I993» 1996). 



independent sample was selected from a frame of 2,054 PSUs with a population of at least 
20,000 people according to the population projections for 1988. It was a systematic sample 
selected with probabilities proportionate to the square root of the population projections for 1988 
from each of the 16 strata defined above. A minimum of 2 new sample PSUs was allocated to 
each of the 16 strata, and the remaining 32 sample PSUs were allocated to the 16 strata to 
approximate a uniform sampling fraction for selecting PSUs in each stratum. One PSU was 
sampled in both the overlap and the independent components, resulting in a final sample of 123 
PSUs. Weighting adjustments were used in the analysis to compensate for the duplicate 
selection. 

Round 3 used the 1991-92 PSS area sample that consisted of 123 PSUs: 8 certainty PSUs as 
before, 58 PSUs that were in the PSS for the first time in 1990-91, and 58 PSUs selected 
independently. The sample overlap with the 1989-90 PSS was kept at about 50 percent. The 
independent sample of PSUs was selected using the same frame, measure of size, and sampling 
method as the 1989-90 PSS. One PSU was again sampled in both the overlap and the 
independent components resulting in a final sample of 123 PSUs for Round 3. 

Area frame constniction. Within each PSU sampled for the PSS, regional staff of the U.S. 
Census Bureau constructed lists of private schools from the following sources: telephone book 
yellow pages, non-Roman Catholic religious institutions, LEAs, chambers of commerce, and 
local government offices. The 1983 PSS also used lists from commercial milk companies and 
commercial real estate offices. Roman Catholic religious institutions were not contacted because 
the list frames provided good coverage of these schools. An area canvas (a block-by-block 
listing of all private schools) was not attempted in these operations. 

For SASS, schools on the lists constructed in the sampled areas are matched with the schools on 
the updated list frame for those areas. Schools not found on the list frame were contacted by 
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telephone and retained in the area frame if they were found to be in-scope for SASS. The area 
frame consisted of 996 schools in Round 1, 900 schools in Round 2, and 355 schools in Round 3. 

Table 2-4 summarizes the private school frame and sample sizes for the list and area frames for 
Rounds 1 to 3. The total frame count is the sum of schools in the list frame and a weighted 
estimate from the area sample of the number of schools not in the list frame. The high Round 1 
total frame count may be due to some erroneous inclusions since this was the first SASS private 
school frame. The number of private schools estimated from Rounds 2 and 3 are comparable at 
about 26,000 schools. The area component accounted for about 24 percent of all private schools 
in the United States in Round 1, 21 percent in Round 2, and dropped to 5 percent in Round 3 
because of improvements in the list frame. 

Table 2-4. — Sample sizes for private schools in the School Survey, by type of frame; 

Rounds 1-3 



Private school frames and samples 


Round 1 
(1987-88) 




Round 2 
(1990-91) 




Round 3 
(1993-94) 




Number Percent 


Number Percent 


Number Percent 


Total frame size 


31,848 


100 


25,998 


100 


26,093 


100 


List frame 


24,186 


76 


20,600 


79 


24,767 


95 


Area frame (weighted estimate) 


7,661 


24 


5,398 


21 


1,326 


5 


Total sample size 


3,513 


100 


3,270 


100 


3,360 


100 


List sample 


2,968 


85 


2,670 


82 


3,202 


95 


Area sample 


551 


15 


600 


18 


158 


5 



SOURCES: NCES (1991a); Gruber, Rohr, and Fondelier (1993, 1996). 



The private school samples were selected independently from the list and the area frames. The 
sample sizes from the two frames are shown in table 2-4. The sampling fractions for the list and 
area frames were comparable, at least for Rounds 2 and 3. 

List samples. To select the list sample for each round, schools were allocated to strata. Round 1 
used strata defined by state, association group, and school level. If an association group had less 
than 100 schools, then all schools in the association group were included in the sample with 
certainty. The remainder of the private school sample was allocated proportional to the number 
of schools in the association group. Within a state and school-level stratum, a raking procedure 
was used to distribute the private school allocation to the association groups. Kaufman (1991, 
table 5) provides the numbers of schools in the private school list frame by association group and 
school-level strata. A systematic sample of schools was selected from each stratum with 
probability proportionate to the square root of the QED number of teachers in the school. 
Schools with a measure of size exceeding the sampling interval were included in the sample with 
certainty. The procedure to control for sample overlap with other NCES surveys was built into 
this part of the selection process (Kaufman, 1991). 
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For Rounds 2 and 3, strata were defined by association group, school level, and census region. 
Within each stratum, schools were sorted by state and several other variables within state (e.g., 
highest grade, urbanicity, ZIP Code, and student enrollment). A specified number of schools 
was selected from each stratum systematically with probability proportionate to the square root 
of the number of teachers as reported in the frame (see Kaufman and Huang, 1993, for Round 2; 
and Abramson et al., 1996, for Round 3). All schools whose measures of size exceeded the 
sampling interval for the stratum were selected with certainty. For the remaining schools, the 
controls on overlap between rounds of the SASS School Survey were built into this part of the 
selection process. 

Private school association groups. Table 2-5 shows the sample allocation of the private schools 
from the list frame by association group for Rounds 1 to 3. Round 1 included 13 association 
groups, and Rounds 2 and 3 included 18 and 19 groups respectively. All schools in association 
groups with fewer than 100 schools were included; the association groups involved included the 
Association of Military Colleges and Schools, Friends Council on Education, Solomon Schechter 
Day Schools, and some Lutheran Church schools. For other association groups, the minimum 
sample size was about 100 schools. Some association groups were sampled at relatively high 
rates. For example, 88 percent of schools from the National Independent Private School 
Association were sampled in Round 3. 

Area samples. The area samples in each round included nearly all unlisted schools in selected 
PSUs. In most states in Round 1, all area frame schools were included in the sample. In six 
states (California, Florida, Illinois, Montana, Nevada, and Texas) schools were sampled to avoid 
area sample sizes exceeding 30 percent of the state's total allocation. 

The Round 2 area sample included all 453 schools found in noncertainty PSUs and a sample of 
147 schools selected from the 447 schools found in certainty PSUs. The 447 schools were 
stratified by level and then sorted by association group, enrollment, and alphabetical order of 
school name, and the 147 schools were selected with probabilities proportionate to the square 
root of the reported number of teachers. There were no controls for overlap with the Round 1 
sample schools. The Round 3 area sample included all 158 schools listed in the noncertainty 
PSUs. The 197 schools listed in certainty PSUs were added to the PSS list frame before SASS 
sampling. 

Design effects. The complex sampling design of SASS produces sampling variances different 
from those produced by a simple random sample of the same sample size. The impact of the 
complex design on the reliability of a sample estimate, relative to a simple random sample, can 
be measured by the design effect (deff). In general, this measure reflects increases in the 
variance arising from the clustering and departures from an equal probability sample and 
decreases from the use of stratification and the estimation procedures. In the case of the school 
survey, since there is no clustering, the measure reflects only the other factors. A deff larger than 
1 indicates that the sampling error is higher than that from a simple random sample of the same 
size. 

Salvucci, Weng, and Kaufman (1995) and Salvucci, Holt, Moonesinghe, and Kaufman (1995) 
computed deffs for groups of statistics by characteristics for the School, the School Principal, the 
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Table 2-5. — Sample allocation of private schools from the list sample and percent of schools 
sampled, by school association group: Rounds 1-3 



Association group 

Total sample from list frame 

Association of Military Colleges and Schools 
Friends Council on Education 
National Catholic Education Association, and 
Jesuit Secondary Education Association 
National Association of Episcopal Schools 
National Society of Hebrew Day Schools 
Solomon Schechter Day Schools 
Other Jewish 

Lutheran Church — ^Missouri Synod 
Evangelical Lutheran Church — ^Wisconsin 
Evangelical Lutheran Church in America 
Other Lutheran 

General Conference of Seventh-day Adventists 
Christian Schools International 
American Association of Christian Schools 
International 

National Association of Private Schools for 
Exceptional Children 

American Monte ssori Society Schools, other 
Montessori schools 

National Association of Independent Schools 
National Independent Private School 
Association 
Other private schools 



Sample size Percent of schools sampled 

Round 1 Round 2 Round 3 Round 1 Round 2 Round 3 

(1987-88) (1990-91) (1993-94) (1987-88) (1990-91) (1993-94) 



2,968 


2,670 


3,202 


9 


10 


12 


26 


20 


33 


100 


100 


100 


63 


67 


85 


100 


100 


100 


781 


716 


930 


9 


8 


11 


120 


98 


100 


42 


33 


27 


1 


101 


121 1 


1 ■ ■ 


68 


47 


f 142 


55 


57 


^25 


100 


100 


1 


100 


100 J 


1 


35 


24 


j 


100 


100 1 


1 


10 


9 


> 222 


100 


99 1 




35 


26 


93 


100 


100 


84 


) 


98 


60 ^ 


1 


51 


100 


195 


103 


102 


15 


9 


9 


118 


101 


133 


46 


35 


14 


Ml 


99 


113 


17 


11 


12 


110 


100 


177 


72 


54 


63 


121 


103 


101 


41 


25 


15 


172 


218 


269 


18 


26 


30 


* 


* 


100 


* 


* 


89 


721 


398 


422 


9 


7 


5 



*Not applicable. (Not listed as a separate association group.) 

SOURCES: NCES (1991a); Gruber, Rohr, and Fondelier (1993, 1996). 

Teacher Demand and Shortage Survey, and the Teacher Survey in Round 2. Table 2-6 shows 
average deffs for the School Survey by sector for groups of national estimates relating to student 
totals (e.g., number of students enrollment in first grade), teacher totals (e.g., number of full-time 
teachers), student averages (e.g., average number of ungraded students), teacher averages (e.g., 
average number of Hispanic teachers), and school proportions (e.g., proportion of schools 
offering kindergarten). 

The average deffs in table 2-6 all exceed 1, indicating that the survey estimates are less precise 
than the estimates that would have been obtained from a simple random sample of the same size. 
This loss of precision for national estimates occurs because of the compromises made in 
designing the school sample to meet the sample’s multiple objectives, in particular the 
disproportionate allocation of the sample across domains to produce domain estimates of 
adequate precision, and the use of the square root of the number of teachers as a measure of size 
for sampling schools in order to also satisfy the needs of the Teacher Survey. 
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Table 2-6. — Average design effects for selected estimates: Round 2 







Average design effect (deff)* 








Numbers 


Averages 




Proportion 


Sector 


Students 


Teachers 


Students Teachers 


Schools 


Public schools 


1.79 


1.48 


1.58 


1.41 


1.74 


Private schools 


1.11 


1.27 


1.18 


1.19 


2.05 


Number of estimates 


34 


28 


34 . 


28 


13 



♦Design effects were not available for Rounds 1 and 3. 
SOURCE: Salvucci et al. (1995). 



2.4.3 Coverage issues and frame accuracy 

Two quality-related features of a sampling frame are its coverage properties for the survey's 
target population and the accuracy of information it provides for individual units. This section 
summarizes the results of several evaluation studies of these issues. 

Omissions. For Round 1, the QED was found to have excluded 275 small LEAs in Nebraska 
with schools in elementary grades only. The missing LEAs contained only 2,800 students. 
Since these LEAs were excluded from the QED, they were not represented in the SASS 
estimates, and the schools associated with these LEAs also were not represented in the SASS 
school sample for Round 1 (see Hammer and Gerald, 1991, p. 22). 

Erroneous inclusions. At each round some sampled schools were found to be out of scope for 
the School Survey (e.g., schools that did not hire teachers). Such schools were dropped from the 
samples, thus resulting in slightly smaller sample sizes than planned. For example, in Round 1 
some schools that did not hire teachers were left in the QED frame in Louisiana, Arizona, and 
Virginia. In Round 2, the sample losses due to out-of-scope cases comprised 4 percent of the 
public school sample, 6 percent of the private school sample, and 1 percent of the BIA school 
sample. The sample losses in Round 3 were similar at about 3 percent for the public school 
sample and 8 percent for the private school sample (Monaco, Salvucci, Zhang, Hu, and Gruber, 
1998, table 6.2.2). . 

School definition problems: Some problems occurred in all three rounds of the School Survey 
when the listing of schools on the CCD, QED, and PSS did not conform to the definition of 
schools used in SASS. In Round 2, for example, special education programs in California were 
found to have been listed on the CCD as a single school entity; in fact, SASS treats each location 
offering a special education program as a separate school. Thus, the Los Angeles special 
education program was listed on the CCD as one school but actually consisted of 115 separate 
locations, 74 of which were at regular schools already included separately on the CCD and an 
additional 41 locations that did not appear at all on the CCD. This same problem also occurred 
in Illinois and Pennsylvania. Once identified, lists of sites for the multisite programs were 
obtained and matched against the CCD (or other files), and a sample of those not in schools 
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already included on the CCD (or other files) was selected and included in the survey (Kaufman 
and Huang, 1993, chapter 10). 

The problem also was found to have recurred in Round 3. Again, the Los Angeles special 
education program was listed as one school on the CCD when, in fact, the program was located 
in a total of 136 separate locations, of which 30 were found not to be on the CCD file. As in 
Round 2, a universe file of all locations for all special education programs in California was 
developed, compared against the CCD file, and samples selected to represent those program 
locations that were not in schools already included on the CCD (Abramson et al., 1996). 

Frame and survey reference periods. Another factor that may affect coverage is the difference 
between the period for which the sampling frame was constructed and the reference period for 
the survey. For instance, the public school sample for Round 3 of SASS was selected from the 
CCD for school year 1991-92, whereas the reference period for the School Survey was school 
year 1993-94. Schools beginning operation after school year 1991-92 were not covered unless 
they resulted from a split of an existing school or a merger involving one or more existing 
schools. Using data from the 1993-94 CCD, it would be possible to identify schools that were in 
operation in school year 1993-94 but were not included in the sampling frame for Round 3 of 
SASS. However, this operation is complex because not all CCD schools are eligible for SASS; it 
has therefore not been done. 

Similarly, the private school list sample frame for the Round 3 1993-94 School Survey was one 
that was developed for the 1991-92 PSS, so that schools that started operation after spring 1992 
would normally not be included in the list frame. To partially address this problem. Round 3 
updated the 1991-92 PSS for 1993-94 by using lists obtained from association groups in early 
1993. 

Public school district and school coverage. NCES has conducted several studies to evaluate the 
CCD coverage and quality (Owens and Bose, 1997; Owens and Young, 1999; Hamann, 2000; 
Lee, 2000). The CCD coverage of LEAs and schools are linked because, by definition, all 
schools on the CCD list are linked to an LEA. However, there are some LEAs with no 
associated schools. 

At the LEA level, Owens and Bose (1997) found that the coverage of the 1994-95 CCD agency 
list was fairly complete for regular traditional LEAs providing elementary and secondary 
education. However, the coverage of nonregular LEAs that provide nontraditional services, such 
as charter schools and vocational-technical schools, was more problematic. This evaluation was 
based on comparisons of the CCD lists of agencies and schools with several other sources 
including: the Government Integrated Directory (GID), state education directories, the Integrated 
Postsecondary Education Data System (IPEDS), BIA Directory, the DOD Directory, and the 
1993-94 CCD agency list. Owens and Bose found that agencies in other sources and not in the 
CCD list constituted about 5 percent of the total number of agencies on the CCD list. Most of 
the omitted agencies were found on the state education directories, and many of the omissions 
were found in California, Kansas, Massachusetts, Minnesota, Oklahoma, Texas, and Wisconsin. 
The CCD agency coverage was complete in five states (Delaware, Hawaii, Mississippi, 
Nebraska, and Nevada). 
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With regard to school coverage, the CCD public school universe does not provide universal 
coverage of all BIA schools. Owens and Bose (1997) found that of the BIA schools listed on the 
BIA Program Education Directory, only about 25 percent were covered by 1994-95 CCD. Some 
states listed BIA schools as private schools and reported them to the PS S (these schools were 
dropped from the PSS before the survey was conducted). Among schools operated by the DOD 
in military bases, omissions were found for two DOD agencies: the Antilles Consolidated School 
System in Puerto Rico and the West Point School in New York. Many DOD schools reported to 
the CCD by state agencies had no data reported for them. A detailed report of CCD school 
coverage is being prepared and should be released in 2000. 

Hamann (2000) compared the coverage of 1994-95 CCD school universe file with two databases 
prepared by private firms: the QED and the Market Data Retrieval (MDR). This study matched 
the three universe files in turn to identify schools found in one source but not in the others. A 
draft report of this study is available and preliminary results show that the CCD file provides 
good coverage of regular schools. 

Some insights about the coverage and quality of the CCD and QED as public school sampling 
frames are provided by a study conducted by Westat through the Education Statistics Services 
Institute (ESSI) for NCES, which evaluated the quality of these two files for the purpose of 
creating a sampling frame to select school samples (Lee, 2000). The study compared the QED 
(March 1997 edition) and the CCD (1995-96 school year) files against the 1998 NAEP public 
school sample. The NAEP sampling frame was based on the QED, but data on the sampled 
schools also incorporated the results of the NAEP field management system (FMS). The I^S 
collects field data from sample schools on eligibility (e.g., open or closed, regular or not regular) 
and data from participating schools on schools contact information (e.g., phone number and 
address) and school size (e.g., student enrollment). This adjusted sample was considered as the 
standard, against which to evaluate both the QED (the frame used to select the sample schools) 
and the CCD (the specifications for linking NAEP and CCD schools were provided by 
Dymowski, 1999). The results showed the QED somewhat more up-to-date than the CCD in the 
accuracy of school contact information, and revealed a small amount of undercoverage in CCD. 
This may occur, in part, because of the operational differences in the availability of the two 
files — CCD files are annual files, whereas QED files are updated almost on a continual basis and 
released monthly. In terms of student enrollment, however, the CCD enrollment data provide a 
slightly better prediction of the field observed grade enrollment than the QED enrollment data. 
This evaluation is limited somewhat in that the NAEP sample covers only schools with grades 4, 
8, and 12 and, further, the study could not address issues of QED undercoverage. When the two 
frames were compared in full,, Hamann (2000) found undercoverage in both frames, and 
concluded that both the CCD and QED frames provide comparable coverage of regular public 
schools. This study, however, lacked any external verification of the existence and status of the 
listed schools. 

SASS has used the CCD frame to select the sample of public schools beginning with Round 2; 
given the improvements introduced into the CCD process, along with earlier availability of the 
file, there would appear to be no advantage to SASS in using a QED frame. 
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Private school coverage. The estimated proportions of schools from the list and area frames in 
table 2-4 provide an indication of the completeness of coverage of the list frames. As noted 
earlier, in Rounds 1 and 2 the area frames accounted for more than 20 percent of the estimated 
total number of private schools in the combined frames (NCES 1991a, p.9; Kaufman and Huang, 
1993). Data for Round 2 show that the estimated proportion of schools coming from the area 
frame varied substantially by association group. The area frame contribution was especially 
large for schools that were members of the National Association of Private Schools for 
Exceptional Children (35.4 percent) and the American MOntessori Schools Society (31.6 
percent). In contrast, the area frame accounted for less than 5 percent of schools associated with 
the Friends Council on Education, the Association of Military Colleges and Schools, and 
Christian Schools International (Kaufman and Huang, 1993, appendix 4, table 18.) The area 
frame contribution dropped to 5 percent in Round 3 because of improvements to the list frame. 

The updating of the list frame for the 1991-92 PSS used state lists of private schools as well as 
association lists and other lists. About 80 percent of the schools added to the list frame during 
the update came from the state lists (Jackson, Frazier, King, and Schwanz, 1994). The private 
association lists contributed about 1 1 percent and other lists about 9 percent of the additions. By 
school characteristics, the list update had the largest impact in improving the coverage of 
nonsectarian schools, non-Catholic religious schools, elementary schools, and small schools. 

Two studies, Jackson and Frazier (1996) and Jackson, Johnson, and Frazier (1997), examined the 
coverage of the 1993-94 and 1995-96 PSSs and evaluated the contribution of the list updating 
activity and the area search component in improving private school coverage. Using capture- 
recapture methodology the PSS coverage of private schools in both years was estimated to be 
about 98 percent. The area search activities were more expensive than the list updating 
activities. However, since the area sample component represented about 8 percent of the schools 
covered in the PSS, it is needed to provide adequate coverage of private schools. 

In Round 1, the private school list frame contained duplicate listings for some schools, usually 
with slight differences in the name or address of the school. Those discovered prior to sample 
selection were removed during frame construction. Some were discovered after sample 
selection; these schools received a weighting adjustment to account for their increased 
probability of selection (NCES, 1991a, p.8). Area frame schools that were on the list frame were 
removed from the area fr^e before sampling and there were no reports of duplicates between 
the list and area frames being discovered after sample selection. 

Two recent studies evaluated the PSS and the QED as list frames for the selection of samples of 
private schools. Green (2000) constructed a combined PSS and QED frame for use in the 1996 
NAEP. The combined frame improved coverage over either the PSS or the QED individually. 
The relative gains were marginal and a combined frame was not used for subsequent NAEP 
samples, both because of concerns about possible undetected duplication resulting from the 
merger of the two lists, and because the extensive resources required to merge the files correctly 
appeared unjustified. A subsequent study, evaluating the 1998 PSS and QED frames suggested 
that the PSS was superior (see Burke, 2000), as QED showed a degree of undercoverage. Both 
the 1998 and 2000 NAEP samples of private schools were selected from PSS. 
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Classification issues. Occasionally, public schools have been included mistakenly on the private 
school lists received from some states. As mentioned above, some BIA schools have been 
reported as private schools to the PSS. These schools were removed from the PSS list in 
constructing the private school survey list. 

In some instances, the private school frames contain incorrect information on school 
characteristics that were used in the sample selection process. In Round 1, for example, some 
private schools in the School Survey reported an association membership different from that 
recorded on the frame and used for sampling. Some schools reported being members of the 
Friends Council on Education, Association of Military Colleges and Schools, or Christian School 
International association groups, whereas on the frame they were' listed as belonging in other 
groups that had been sampled at a much lower rates (all schools on the list frames for the Friends 
Council on Education and the Association of Military Colleges and Schools were selected with 
certainty). No bias was introduced into the estimates for the affected association groups, but 
their sampling errors were substantially increased (Kaufman and Huang, 1993, appendix 4). 

Missing data on the frame. Information on the number of teachers and enrollment was 
sometimes lacking for schools on both the public and private list frames for each round. Because 
teacher counts were needed to determine selection probabilities, they were imputed for these 
schools. In Rounds 2 and 3, values were imputed from the previous round of SASS, when 
available, from the application of assumed student-teacher ratios to enrollment figures, or by 
using the median value for other schools in the same stratum. The use of imputed values in place 
of reported values is likely to result in some loss in sampling efficiencies. There is no 
documentation of the effects of imputation on sampling efficiencies in any round. 

2.5 SURVEY CONTENT 

The School Survey collects information on many aspects of elementary and secondary schools. 
The core component of the survey pertains to basic school characteristics, staffing patterns, 
programs and services, and school policies. Some examples of the data collected in the Round 3 
school questionnaires include enrollment by grade level, students by race/ethnicity, number of 
male students, number of absent students on the most recent school day, admission requirements, 
type of school, number and types of staff, number of teachers by race/ethnicity, number of absent 
teachers, teaching vacancies, programs and services offered (magnet programs. Chapter 1 
services. National School Lunch Program, remedial reading, remedial math, programs for 
students with disabilities, programs for gifted and talented students, day care, English as a 
Second Language, bilingual education, diagnostic and prescriptive services, health care services, 
library, prekindergarten programs, alcohol and drug use prevention programs and counseling), 
and for high schools, the number of 1993 graduates and the number of 1993 graduates who 
applied to colleges. 

Survey Forms. Round 1 used two questionnaire forms, one for public schools (Form SASS-3A) 
and one for private schools (Forms SASS-3B). Most items were the same on the two forms, with 
some items specific to private schools. With the addition in Round 2 of a separate sample of 
BIA schools, a BIA school questionnaire (Form SASS-3C) was added for BIA schools operated 
outside the local public school systems. Further, since Round 2, the private school and BIA 
school questionnaires have incorporated items from the TDSS questionnaire that is addressed to 



LEAs in relation to public sector schools administered by LEAs. Round 3 also used three 
separate questionnaires for public, private, and BIA schools. The Round 4 School Survey 
includes for the first time all charter schools in operation as of 1998-99. A fourth school 
questionnaire form has been added for charter schools to collect additional information on when 
the charter was granted and by whom, what types of regulations were waived and their 
importance, whether the school is new or was converted from a pre-existing school, and whether 
the school operates within a school district. 

Changes between rounds. The SASS Data File User's Manuals fNCES, 1991 a, b, c, d; Gruber, 
Rohr, and Fondelier, 1993 and 1996) document the changes in questionnaire items in each round 
of the survey. Section 2.6.2 in this report summarizes the cognitive research used to improve the 
questionnaire and its design. 

The subject content for the questionnaires used in SASS reflects user needs as determined 
through interactions with data user groups, technical review boards, education policy researchers, 
and experts in the fields of teacher training and utilization. Continued interaction with the user 
groups also contributed to providing insight into new data needs at each round. The various 
groups involved in the survey content development include the SASS Private School Users 
Groups; the SASS Technical Review Board; the National Academy of Sciences; the National 
Forum on Education Statistics; the National Education Goals Panel; the Council of Chief State 
School Officers; and the National Science Foundation. 

In 1999-00, the Round 4 questionnaires for public, private, and BIA schools have incorporated 
new items on; computers (number, access to the Internet, and whether there is a computer 
coordinator in the school); availability of certain types of curricular options; how special 
education students' needs are met; changes in the school year or weekly schedule; the school's 
enrollment capacity and whether the school has a program for disruptive students. 

2.6 DATA COLLECTION 

Data collection for the School Survey is accomplished through mailout and mail-return 
questionnaires with telephone followup of nonrespondents. This section discusses the data 
collection and the associated supervision and quality assurance procedures (2.6.1); the results 
obtained from cognitive research and pretests for each round of the survey (2.6.2); reinterview 
programs to measure response variance (2.6.3); and nonresponse (2.6.4). Measurement errors 
detected during the data file preparation and processing stages of the survey are discussed in 
section 2.7. A report. Measurement Error Studies at the NCES (Salvucci et al., 1997), 
summarizes various measurement error studies conducted for SASS and other NCES surveys. 

2.6.1 Data collection procedures 

The School Survey is conducted by mail with telephone followup of schools that fail to return 
the mailed questionnaires. The questionnaires are addressed to school principals who are asked 
to complete and return them to the U.S. Census Bureau within 3 weeks. There are no restrictions 
on who should complete the questionnaires; principals who wish to do so can assign the task to 
someone on their staff. Since response to the SASS is voluntary, the data collection procedures 
include a number of features to encourage cooperation. This section summarizes the data 
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collection activities, the mode effects in data collection, supervision of field representatives and 
quality assurance, and response burden. 

Data collection activities. The general pattern of activities has remained more or less the same 
across rounds, but in later rounds, the schedule of activities has started earlier in the school year 
in an attempt to improve the mail-return response rate and to allow time to complete all the 
SASS components in the school year (see table 2-7). The process starts with the mailing of 
advance letters to the sampled schools and the LEAs of sampled public schools to provide 
general information about the survey and to ask for cooperation. The advance letter to the 
schools also asked them to submit lists of their teachers, to be used in selecting teachers for the 
Teacher Survey. The first mailing of the school questionnaires took place in January of the 
school year for Rounds 1 and 2. This mailing was moved forward to December of the school 
year for Round 3 and to September for Round 4. A postcard reminder sent 1 week after the first 
mailing was introduced in Round 3 and was retained in Round 4. A second mailing of school 
questionnaires was sent a month after the first mailing to schools that had not yet responded. 
Finally, mail nonrespondents were followed up by telephone as described below. 



Table 2-7. — Schedule of data collection activities for the School Survey: Rounds 1-4 



Activities 


Round 1 
(1987-88) 


Round 2 
(1990-91) 


Round 3 
(1993-94) 


Round 4 
(1999-00) 


Advance letters to 
LEAs 


Fall, 1987 


September, 

1990 


August, 1993 


August, 1999 


Advance letters to 
schools 


Fall, 1987 


September, 

1990 


September, 

1993 


August, 1999 


First mailing of school 
questionnaires 


January, 1988 


January, 1991 


December, 

1993 


September, 

1999 


Postcard reminder 


(not used) 


(not used) 


One week ^ter 
first mailing 


One week after 
first mailing 


Second mailing of 
school questionnaires 


February, 

1988 


February, 

1991 


January, 1994 


October, 1999 


Telephone followup 


April-June, 

1988 


March- June, 
1991 


January-June, 

1994 


November, 
1999- June, 
2000 



SOURCES: NCES (1991a); Gruber et al. (1993, 1996); NCES (1999). 



Telephone followup of mail nonrespondents. Telephone followup activities for schools that had 
not responded to the second mailing were initiated within about 3 weeks after mailing the 
questionnaires. In Rounds 1 and 2, U.S. Census Bureau field representatives, working from U.S. 
Census Bureau regional offices in two regions and from their homes in the other regions, 
attempted to complete the questionnaires by telephone. They were instructed to try to reach 
school principals during normal working hours between 8:00 a.m. and 5:00 p.m, to offer to hold 
the line a few minutes while respondents checked school records, to call back, and to accept 
collect calls from respondents (U.S. Census Bureau, 1991b). Respondents who were unwilling 
to participate in telephone interviews were asked to return the mailed questionnaire. 
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Telephone followups of public schools in Round 3 used computer-assisted telephone 
interviewing (CATI) with the interviews being conducted by U.S. Census Bureau field 
representatives from two U.S. Census Bureau telephone centers. In this round, about 60 percent 
of the sample public schools received the second mailing of questionnaires. Table 2-8 shows the 
number and percent of sample schools requiring telephone followup and the number and 
percents of interviews completed. Telephone followups of private and BIA schools were 
conducted by field representatives completing paper copies of the questionnaires as in previous 
rounds. There were 41 percent of private schools and 44 percent of BIA schools that required 
field representative followup, and the followup successful completion rates were 67 percent and 
99 percent for these two school types, respectively (Gruber et al., 1996). 



Table 2-8. — Schools requiring and completing telephone interview followup, by type of school 
and followup interview: Round 3 



School type 




CATI followup 






Non-CATI followup 




Total 


Completed 




Total 


Completed 




Number 


Percent* 


Number Percent'* 


Number 


Percent'* 


Number Percent'* 


Public 


4,284 


44 


3,111 


73 


382 


4 


362 


95 


Private 


t 


t 






1,385 


41 


932 


67 


BIA 


i 








71 


44 


70 


99 



*Not applicable. 

‘Percent of sample cases in CATI followup. 

^Percent of sample cases in CATI followup that were completed. Cases identified as noninterview and out of scope for the survey during the 
CATI followup are not included. 

^Percent of sample cases in telephone followup by field representatives. 

^Percent of sample cases in telephone followup by field representatives that were completed. Cases identified as noninterviews and out of scope 
for the survey during followup are not included. 

NOTE: Numbers do not total because of noninterviews and out-of-scope cases and because nonresponse cases may be switched between CATI 
and non -CATI followup. 

SOURCE: Gruber etal. (1996). 



The percent of responses received by mail (mail responses as a percent of mail plus telephone 
responses) varied substantially among different subgroups of the school universe and by round. 
Parmer, Shen, and Tan (1992) report that in Round 2 mail returns accounted for 67 percent of all 
responses for public schools and 56 percent for private schools in the list frame. The percent of 
responses received by mail varied widely by state and. metro status for public schools, from 48 
percent for the District of Columbia to 81 percent for Delaware, and from 55 percent in large 
cities to 74 percent in nonmetropolitan statistical area large towns. The corresponding percent 
for private schools varied by school level and association group, ranging from 48 percent for 
combined schools to 60 percent for elementary schools, and from 31 percent for the American 
Association of Christian Schools to 74 percent for the Lutheran, Missouri Synod schools (see 
Parmer et al., 1992, tables A1-A4). 



In Round 3, 56 percent of responses were received by mail for public schools and 65 percent for 
private schools (Monaco et al., 1998), a reverse from Round 2. Since telephone followup 
collected a substantial amount of data, Monaco et al. (1998) noted a concern about possible lack 
of compatibility of responses due to differences in the mode of data collection. 
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Mode effects. Thus far, there is ho conclusive evidence of a mode effect since no controlled 
experiments have been conducted to compare the quality of mail and telephone response in 
SASS and to evaluate the effects of CATI for the school survey. Two studies have found some 
indications of differences between mail and telephone responses to the school questionnaire. In 
Round 1, the reinterviews with about 10 percent of the sample schools were all conducted by 
telephone. The mentioned responses for counts of students served by special programs, such as 
bilingual education, showed more evidence of "heaping" in multiples of 100 than occurred in the 
responses to the initial interviews. This finding suggests a hypothesis that telephone respondents 
are less likely to refer to records or to arrive at a carefully considered estimate than those who 
respond by mail (Bushery, Royce, and Kasprzyk, 1992). 

In a review of information relevant to mode effects in the Round 2 School Survey, Parmer et al. 
(1992) found some evidence of mode differences for private schools. For example, item 
response rates were found to be higher with mail response. The mail response rates show great 
variation by affiliation group (see table 2-9). Mail response rates close to 60 percent or more 
were found in Lutheran, Catholic, Military, and Christian Schools International groups. Rates of 
45 percent or less were found in the following groups: Jewish, Friends, and American 

Association of Jewish schools. However, since the study was not a controlled experiment, it 
could not produce conclusive results about mode effects, nor was it able to indicate which mode 
yielded more accurate information. 

Table 2-9. — Private School Survey mail response rates, for the list frame, by association group: 
Round 2 



Association group 


Percent obtained by mail* 


Total 


55 


Association of Military Colleges and Schools - United States 


67 


Catholic 


63 


Friends 


42 


Episcopal 


51 


National Society for Hebrew Day Schools 


35 


Solomon Schecter 


43 


Other Jewish 


36 


Lutheran - Missouri Synod 


74 


Evangelical Lutheran Church - Wisconsin Synod 


66 


Evangelical Lutheran Church in America 


71 


Other Lutheran 


58 


Seventh-day Adventists 


57 


Christian Schools International 


64 


American Association of Christian Schools 


31 


National Association of Private Schools for Exceptional Children 


58 


Montessori 


49 


National Association of Independent Schools 


49 


Other private schools 


50 



♦Mail responses as a percent of mail plus telephone responses. 
SOURCE: Parmer, Shen, and Tan (1992). 



Supervision of field representatives and quality assurance. The field representatives who 
conducted the telephone interviews at each round were experienced survey interviewers who had 
already been trained in basic interviewing procedures and concepts (e.g., confidentiality and how 
to persuade reluctant respondents to participate.) They were given an instruction manual for 
SASS and were trained on the content and procedures for the survey by a self-study training 
package. The field representatives mailed their completed questionnaires to their regional 
offices on a flow basis, where the forms underwent review and clerical edit. For each field 
representative, the first two questionnaires received were reviewed for errors by regional office 
staff. If a total of 10 or more errors was found in the two questionnaires, the field representative 
was notified of the errors and given suggestions for improvement. This process was repeated for 
successive sets of two questionnaires until the field representative succeeded in completing a set 
with fewer than 10 errors (Gruber et al., 1993). 

In Round 2, the U.S. Census Bureau collected the Field Representative's Edit and Performance 
Records (SASS Form-23) from the regional offices. A review of these forms found that most of 
the regional offices had not completed them correctly and one office apparently had not used 
them (Pasqualucci, 1991). The review also found that many of the field representatives had 
accumulated more than 10 errors on their first four questionnaires. The common errors were 
failures to follow skip patterns and failure to follow instructions to check "None" boxes rather 
than entering a "0" response. In an item on the public school questionnaire asking for a 
breakdown of enrollment by race/ethnic category, the total for all categories frequently was not 
equal to the total enrollment reported in a prior question. Some field representatives had entered 
a percent instead of a whole number for each category. With private school questionnaire items 
requiring decimal entries (e.g., years of instruction required for graduation by subject) and items 
relating to full-time equivalent (FTE) staff, some field representatives failed to record any digits 
to the right of the preprinted decimal points. 

In 1992, the U.S. Census Bureau tested the development of a prototype automated data collection 
instrument for use in one or more of the SASS surveys. Schools or other units willing to use this 
mode would receive a diskette containing the survey questionnaire and instructions for 
completing it. Using their own microcomputers, they would enter their responses on the diskette 
and return it to the U.S. Census Bureau. There was some small-scale testing of the prototype in 
Round 3 with limited success. As a result, this mode of data collection was not implemented. 

The Round 3 CATI interviews for public schools were conducted by interviewers at two 
centralized telephone interviewing facilities. The CATI interviewers were provided with an 
instruction manual and were trained on the survey content and procedures. The computerized 
program included some built-in data edits, and data collected by CATI were transferred 
electronically to a central site for further processing (see Gruber et al., 1996). 

Length of time to complete questionnaire. To estimate response burden, the Round 2 private 
school questionnaire included an item about time spent in completing the questionnaire. For 
questionnaires returned by mail, the person who completed the questionnaire provided the 
answer; for questionnaires completed in followup telephone interviews, the interviewer answered 
the item. The median time for completion was 60 minutes, with an interquartile range of 50 
minutes. For about 90 percent of schools, the questionnaire was completed in less than 2 hours 
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and 10 minutes, and for 1 percent, it took more than 5 hours. The item about time burden was 
not asked in the Round 2 public school questionnaire, which was shorter than the private school 
questionnaire. As mentioned above, the Round 2 private school questionnaire was modified to 
include items used in a separate TDSS for public schools. 

Round 3 included the item about time spent in both the public and private school questionnaires. 
However, responses to these items were lost due to a processing error, and no new analysis of 
time burden was conducted for Round 3. 

2.6.2 Cognitive research and pretests 

Before each survey year, NCES conducts extensive cognitive research, field tests and, in 
Round 1, in-depth reinterviews to seek improvements in questionnaires and survey procedures. 
These studies identify measurement errors in the survey, test new or revised questionnaire items, 
and test changes in procedures for the upcoming survey. This section summarizes the pretest 
activities for all SASS components conducted in preparation for the next survey round. Section 

2.6.3 discusses the reinterview studies conducted after each round of SASS for measuring 
response variance. 

Round 1 pretest and in-depth reinterviews. A large-scale pretest was conducted for Round 1 of 
SASS in the early part of 1987. The pretest, which was carried out in 10 states, included 220 
schools, and U.S. Census Bureau interviewers reinterviewed 98 of them by telephone. These 
reinterviews called for in-depth discussions with respondents about how they had arrived at their 
initial answers, and what they had included in their counts and what was excluded, a procedure 
not used in subsequent reinterview programs. Respondents were also asked for their 
recommendations for improving any of the questionnaire items. 

The report of the reinterviews (Nash, n.d.) included several recommendations for improvements 
in specific questionnaire items, many of which were implemented. Certain other items that were 
found to be difficult for some schools to report were subsequently dropped from the 
questionnaire. The following are some examples of the recommendations: for an item on 

special programs, clarify the definition of bilingual education; for an item on student enrollment 
by grade, , clarify the treatment of students enrolled under Head Start and chapter 1 programs; 
38 percent of the schools did not have records available from which to answer an item asking 
about present activities of teachers who had left the school after the preceding school year, so the 
report recommended further review of nonresponse rates for this item to determine how useful 
the results would be. 

Round 2 preparations. The Round 2 pretest reviewed problems identified in the questionnaires 
used in Round 1 and tested new and revised questionnaire items and forms. The major change in 
Round 2 was to consolidate and integrate two questionnaires, for private schools. The TDSS 
questionnaire (SASS-Form IB) was made a part of the private school questionnaire. Another 
change that affected all questionnaires was to include a list of agencies that endorse the survey. 
This endorsement list was displayed prominently on the front of the questionnaires. For public 
schools the list included: the American Association of School Administrators, the American 
Federation of Teachers, the National Association of Elementary/Secondary School Principals, 
and other national and state educational agencies. For private schools, the list included the 



American Montessori Society, the National Catholic Education Association, and other private 
association groups. 

To evaluate the benefits of the changes, questionnaires for Round 2 were mailed to 332 public 
school districts, 352 public schools and principals, 398 private schools and principals, 448 public 
school teachers, and 448 private school teachers. NCES and U.S. Census Bureau staff reviewed 
all the questionnaires returned during the field test to determine the kinds of errors made by 
respondents and to read all comments recorded on the forms. In addition, cognitive interviews 
were conducted in spring 1990 with the school principals of 9 public and 6 private schools, and 
10 public school teachers and 10 private school teachers in the Washington, D.C., area. Trained 
cognitive research staff conducted the interviews, with respondents who were asked to read 
aloud the questions on the form and to verbalize their thoughts as they decided how to answer; 
where necessary, the interviewer asked questions to clarify the respondents' remarks or behavior. 
The interviews were tape recorded for use in subsequent analysis. 

Jenkins and DeMaio (1990) summarized the results of the cognitive interviews with school 
principals who answered the school questionnaires and the school administrator (principal) 
questionnaires. These interviews identified several items asking for counts or percents that were 
difficult for respondents to answer. An item asking for the percent of students enrolled at the 
start of the previous school year who were still enrolled at the end of that year was subsequently 
eliminated from the final version of the public school questionnaire. Two matrix-style items on 
the private school questionnaire that asked for information about FTE teaching staff by grade 
level and subject or specialty proved to be extremely difficult for respondents. They were 
eliminated from the final version of that questionnaire. Other findings from the in-depth 
cognitive interviews led to changes in format and wording for some items. 

Round 3 preparations. The Round 3 preparations for the School Survey included the field test of 
new procedures and questionnaires that occurred in 1991-92 and a cognitive research study of 17 
school principals held in February 1992. The 1991-92 field test took samples of 390 LEAs, 420 
schools, principals, and teachers in the public sector, and 480 schools, principals, and teachers in 
the private sector. This field test evaluated three new procedures, two of which were adopted for 
Round 3. One new procedure introduced in Round 3 was to print a toll-free 800 telephone 
number on the front of each of the SASS questionnaires. Respondents were to use this number 
to call for assistance in completing the questionnaires. There have been no analyses of the use of 
the 800 number. Another new procedure was the mailing of a reminder postcard a week after the 
first mailing of the questionnaire. This reminder encouraged mail return of the questionnaires 
and lessened the need for telephone followups. A procedure that asked for teachers' home 
telephone numbers that would be used to help locate teachers who failed to return the teacher 
questionnaire by mail was not successful in the test and was rejected (see section 5.6.2). The 
SASS Data File User's Manual for Round 3 (Gruber et al., 1996) lists the new questionnaire 
items tested in each questionnaire and the items selected for inclusion in the actual Round 3 
questionnaires. 

A set of in-depth interviews was conducted in February 1992 in 5 mid-western states. The 
interviews, planned for 20 principals, were completed with 17 school principals, 4 each from 
Oklahoma, North Dakota, South Dakota, and Nebraska, and 1 from Iowa (Gruber et al., 1996). 
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The particular states were chosen because they had had high pre-edit failure rates for student and 
teacher counts in Round 2. Three of the four schools in each state had not been in the SASS 
school sample previously; the fourth school was chosen because its School Survey questionnaire 
returned in Round 2 had been rejected at the pre-edit stage due to discrepancies between the 
teacher and student counts and comparable data from the CCD. Interviewing used a condensed 
version of the public school questionnaire that had been used in the 1991-92 field test. The 
purpose was to learn more about questions that had high edit failure rates in Round 2 and to 
provide detailed analyses of new questions that had been developed for possible inclusion in 
Round 3. 

A detailed account of each interview is available, as well as a paper (Jenkins, 1992a) along with 
a memorandum (Jenkins, Ciochetto, and Davis, 1992) that sununarized the main findings and 
provided an item-by-item description of the problems that respondents had in answering the 
questions. Three classes of respondent problems were identified and examples provided. 

One class of problems was respondents’ misunderstanding of concepts. A concept that 
was confusing for several school principals was the school for which they were to report. 
While the name of the school was displayed on the cover page of the school 
questionnaire, some principals who were responsible for multiple schools in small school 
systems either did not notice the school label or were confused about which school to 
report. Among the 17 principals interviewed, eight principals found the information 
about the school unit ambiguous, and three principals actually reported for a school other 
than the one intended. Some of the errors and uncertainties resulted from the 
respondents' failure to give close attention to the label and instructions on the cover page 
of the questionnaire, from a failure to display the name of the school prominently on the 
cover page, and from misleading cues in some of the initial questions. 

Another difficult concept was the full-time versus part-time employment status of staff. 
When respondents were asked to give the numbers of full- and part-time employees in 
each of several instructional and support services categories, they had difficulty in 
correctly classifying employees who had jobs that by their nature could not be full time. 
For example, bus drivers worked part time in more than one category but full time at the 
school, or worked part time at the sample school but full time for the school district. 

A second class of problem concerned format considerations and item layout. A format 
issue was noted with a series of items about limited-English-proficient (LEP) students. 
The introduction to these items included a blank space to record the total number of LEP 
students and a "none" box was placed underneath it with an instruction to skip the 
remaining items on LEP students if there were none. Several respondents entered "0" in 
the answer space, did not notice the "none" box, and proceeded through several questions 
that did not apply to them. Failure to use "none" boxes and follow skip instructions 
correctly is a fairly common problem when questionnaires are self-administered. 

A similar issue of item layout was found with an item asking respondents to indicate the 
level of difficulty they experienced in filling vacancies in a list of fields of instruction. 
The respondents were either to check a box to indicate that they had no vacancies in that 
field or to check one of four boxes by level of difficulty. Some respondents were 
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uncertain how to respond for fields that were not relevant in their schools. The intent of a 
sub-item at the end of this item, asking for specification of subfields in the vocational- 
technical education category, was especially unclear to respondents. 

The third class of problem related to the inefficient use of records. One item on the 
school questionnaire asked about enrollment by grade on October 1 of the current school 
year. Since public schools are required by law to submit such information annually to 
their school district or to their state, the intention was for respondents to use their fall 
official report to complete this item. Instead of retrieving the official report, some school 
principals went through unnecessary work to complete the item and sometimes provided 
erroneous numbers for a different date. 

A related item asked for enrollment by race/ethnicity and listed four categories of 
minority groups and a fifth category of white non-Hispanic students. Many school 
principals reported the numbers of minority students from their records and derived the 
number of white non-Hispanic students by subtraction from the total enrollment on 
October 1 that had been reported in the preceding item. However, in some instances, the 
records used to obtain the data for the minority categories referred to a date other than 
October 1, leading to an erroneous number for white, non-Hispanic enrollment. 

Gruber et al. (1996) further noted that the results of this cognitive research supported a number 
of assumptions that had been made previously about some of the public school reporting 
problems identified in earlier rounds in these states. In some small school districts, the district 
superintendent often acts as the principal of one or more schools and may report for the entire 
district because the information is really available; further, in some of these situations, all the 
schools in the district are located on one campus. Some local school officials and school staff 
who complete the SASS forms also used different school definitions from those used by state 
officials who provide the information for the CCD. For example, Nebraska does not report any 
K-12 schools for CCD; instead such schools are separated into elementary (grades K-6) and 
secondary (grade 7-12). However, some schools containing grades K-12 are located in a single 
building and have only one principal; respondents frequently report all grades in the building as 
one school for SASS. In some areas of Oklahoma, researchers discovered that the term "school" 
refers to the entire district, while "attendance center" is used to designate what is considered a 
school for SASS. 

The research also indicated that several of the proposed new items that asked for counts of 
students enrolled in specific science, math, and computer science classes were too difficult and 
time-consuming to answer and posed definitional problems for some of the class subject name. 
Several respondents said that they were unable to provide accurate responses to these items. 

Round 4 preparations. Jenkins (1997) discusses the cognitive research undertaken to redesign 
the school questionnaires for Round 4. Changes recommended included improving the wording 
for some items; changing the format and overall design; printing the questionnaire on a colored 
background (instead of white) and using white space to highlight answer spaces; reformatting the 
items using a one-column, in place of a two-column, layout to display the questions and answer 
spaces; and using newly designed skip instructions to indicate the paths through the 
questionnaire. These changes were based on recent research that showed that the "look" of a 
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questionnaire could be equally as important to the quantity and quality of the data collected as 
the wording and sequencing of the questions themselves. 

In preparation for Round 4, Cole et al. (1997) reviewed the range of methods used to improve the 
mail return rates, and their applicability to SASS. They note that SASS uses many of the 
recommended methods, including efforts to design user-friendly questionnaires to reduce 
response burden, making multiple contacts with respondents through the mailings of an advance 
letter, the initial questionnaire, a reminder postcard (since Round 3), and if necessary a second 
questionnaire, and the provision of self-addressed prepaid envelopes to facilitate return of the 
completed questionnaires. 

2.6.3 Reinterviews and response variance 

A reinterview program is included as a regular part of the SASS data collection cycle to evaluate 
data quality. Repeat data collections (loosely, reinterviews) are conducted with a subsample of 
the original respondents in order to study the consistency of the responses obtained. In the 
reinterview program, the original data collection procedures are repeated in order to measure the 
reliability of the responses (i.e., response variance), as distinct from the in-depth reinterviews, 
which were conducted in the pretest in Round 1 to investigate the validity of the original 
responses and sources of response error (see section 2.6.2). The SASS reinterview programs for 
Rounds 1 to 3 are discussed in three reports (Newbrough, 1989; Royce, 1994; and Bushery, 
Schreiner, and Sebron, 1998). In addition, Bushery, Royce, and Kasprzyk (1992) compared the 
response variance results for questions reinterviewed in both Rounds 1 and 2, and Conley, Fink, 
Saba, and Kaufman (1997) provide an overview of reinterview programs used by NCES surveys. 

A reinterview study has the limitation that it is difficult to replicate the original interviews. To 
the extent that respondents remember and repeat their answers from the original interviews, the 
reinterviews may not represent independent replications of the original interviews; as a result, the 
reliability of response may be overestimated. On the other hand, the correct responses may have 
changed in the time between the interviews and reinterviews, in which case reliability will be 
underestimated. 

Reinterview sample. The SASS leinterview program for Rounds 1, 2, and 3 has involved 
administering a subset of questions for a second time to a subsample of the original sample of 
schools, school principals, and teachers. Subsamples of library media centers and student 
records were added to the leinterview program in Round 3. This section describes only the 
reinterview program for the School Survey; the reinterview progranis for the other surveys are 
described in later chapters. 

For each round, the reinterviewed sample of schools was a random subsample of about 10 
percent of the original school sample that was selected before the mailout of the main survey 
questionnaires. Reinterviews were undertaken for the sampiled schools that completed the 
original questionnaires. The target was to yield approximately 1 ,000 completed reinterviews of 
schools at each round. 

Table 2-10 shows the sample sizes of the leinterview school samples and the response rates per 
round. While the subsamples of schools were close to target for Rounds 1 and 2, Round 3 had 



only 555 complete reinterviews because of a large number of ineligible cases and a poor 
response rate in this round. About one-third of the ineligible cases in Round 3 were due to 
nonresponse in the original survey; about one-half completed the original survey but were 
deemed ineligible for inclusion (see Bushery et al., 1998). Schools that responded in the field 
followup were ineligible for reinterviews because of timing and other operational considerations. 
The reinterview response rate was poor partly because of the requirement that the same original 
mode of data collection be used for the reinterview. Among schools for mail reinterview, the 
response rate was only 57 percent. For the CATI reinterview cases, the response rate was 74 
percent. 

Table 2-10. — Reinterview sample sizes and response rates: Rounds 1-3 



Category 


Round 1 
(1987-88) 


Round 2 
(1990-91) 


Round 3 
(1993-94) 


Total sampled 


— 


1,123 


1,420 


Number eligible for reinterview 


1,309 


1,034 


899 


Number completed reinterview 


1,138 


940 


555 


Response rate (percent)* 


87 


91 


62 



“Not available. (Data were not collected or not reported.) 

*The response rate was computed using the number of completed reinterviews divided by the number eligible for reinterview. 
SOURCES: Bushery, Royce, and Kasprzyk (1992); Bushery, Schreiber, and Sebron (1998). 



Reinterview procedure. The Round 1 reinterviews for schools were all conducted by telephone, 
using a questionnaire with selected questions from both the school and the school administrator 
questionnaires. The reinterview procedure was changed in Round 2 to better replicate the 
original survey. Thus mail reinterviews were used for mail respondents, and telephone 
reinterviews for telephone respondents, except that when necessary, telephone reinterviews were 
used for mail reinterview cases that failed to respond by mail. The respondent who answered the 
original school questionnaire was asked to complete the Round 2 reinterview questions. 
Round 3 again asked the respondent to the original survey to complete the reinterview and used a 
strict match of mode for the reinterviews. Nonrespondents to the mail reinterview were not 
followed up for telephone reinterview. Of the Round 3 reinterviews, 80 percent were completed 
by the respondent to the original survey, 13 percent were completed by a different respondent, 
and in 7 percent of the reinterviews the name of the reinterview respondent was unknown. 

Reinterview questionnaires that were mailed were typically sent approximately 1 week after 
receipt of the original mail questionnaires. Allowing for time en route, most mail respondents 
received the reinterview questionnaires within 3 to 4 weeks after completing the original 
interview. The telephone reinterviews were typically conducted within 1 to 2 weeks after the 
original interviews. 



Measures of response variance. Two indices have been used in the analysis of the SASS 
reinterview program to measure the degree of inconsistency in responses between the original 
survey and the reinterviews: 
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■ The gross difference rate is the percent of respondents whose responses in the original 
interview and the reinterview were different. 

■ The index of inconsistency measures the percent of total variance for an item that is 
accounted for by response variance. For items with more than two response categories, the 
L-fold index of inconsistency is used; it is a weighted average of the simple index over all 
categories. 

Groves (1989) and Forsman and Schreiner (1991) provide further discussion of these measures 
of response variance. As a rough guideline, response variance is considered to be low when the 
simple or L-fold index of inconsistency is less than 20 percent, moderate when it is between 20 
and 50 percent, and high when it is greater than 50 percent. The gross difference rate is more 
difficult to interpret than the index of inconsistency. A large gross difference rate indicates high 
response variance in the data, but a small rate is no guarantee of good consistency. In a low- 
frequency response category, even a small gross difference rate can represent high response 
variance relative to total variance (Bushery et al., 1998). 

The above indices have been computed in the SASS reinterview program assuming a simple 
random sample of schools. This ignores the unequal selection probabilities and the complex 
sample design used to draw the original school sample from which the reinterview sample was 
drawn. Therefore, these indices may not perfectly reflect the incidence of response error in the 
target population of schools. 

Items evaluated at each round. The items included in the reinterview program were items 
expected to have high response variance. For Round 1, the items included four questions on the 
topics of bilingual education, English as a second language, extended daycare, and the 
community where the school was located. 

Round 2 evaluated more items, including revised versions of the four questions used in the 
Round 1 reinterviews (revised after the Round 1 evaluation) and new items from the following 
four subject categories: student and teacher populations at the school; the school type (regular, 
special education, vocational/technical, and alternative), the community where it was located, 
and the number of days in the school year; programs that the school offered and the grade levels 
of instruction; and teaching vacancies in the school for the year, evaluation programs for 
teachers, and programs to help beginning teachers. 

Table 2-11 shows the indices of response variance for the four questions evaluated in both 
Rounds 1 and 2. There were moderate statistically significant reductions between Rounds 1 
and 2 in the gross difference rates for three of the four questions and in the index of 
inconsistency for two of them. All four questions had indexes of inconsistency in the moderate 
range in Round 2. Changes in reinterview methodology may have contributed to some of the 
reduced response variance, but it seems likely that at least some of the improvement in Round 2 
reflects the revisions made to the questions based on the Round 1 evaluation. 

The Round 3 reinterview program included items on student enrollment by grade; part-time and 
full-time teachers and other staff by grade; student programs and services (e.g., school lunch, and 
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Table 2-11. — ^Response variance of selected items in reinterviews: Rounds 1 and 2 





Percent "yes" response 
(Survey interview) 


Cjtoss difference rate 


Index of inconsistency 


Selected items 


Round 1 
(1987-88) 


Round 2 
(199091) 


Round 1 
(1987-88) 


Round 2 
(1990-91) 


Round 1 
(1987-88) 


Round 2 
(1990-91) 


Bilingual education 


15 


14 


1 

16 


1 

12 


54 


45 


English as a second language 


32 


28 


I 

16 


I 

14 


1 

37 


‘30 


Extended daycare 


16 


23 


9 


9 


32 

1 2 


25 

1 2 


Community where school is 
located 


t 


t 


‘35 


‘30 


■43 


’38 



^Not applicable. 

* Statistically significant difference between Round 1 and Round 2 (at a 10 percent significance level). 

^Values from the Lrfold index of inconsistency. 

SOURCES: Bushery, Royce, and Kasprzyk. (1992). 

vocational-technical programs); and school policies on drug, alcohol, and tobacco use prevention 
programs. 

Table 2-12 summarizes the evaluation results for the reinterview programs in Rounds 1 to 3. 
The index of inconsistency was calculated for items with adequate valid responses for reliable 
calculation (at least 50 cases with valid responses). The results cannot be validly compared 
across rounds because different sets of items were included in different rounds. For Round 3 
over one-half of the items evaluated (70 out of 122) had indices of inconsistency below 20 
percent. 

Table 2-12. — ^Number of school survey reinterview items by index of inconsistency: Rounds 1-3 



Index of inconsistency 


Round 1 
(1987-88) 


Round 2 
(1990-91) 


Round 3 
(1993-94) 


Total questions 


14 


36 


122 


Low (0-19%) 


0 


17 


70 


Moderate (20-50%) 


6 


12 


42 


High (51-100%) 


8 


7 


10 



SOURCES: Bushery, Royce, and Kasprzyk (1992); Bushery. Shreiner, and Sebron (1998). 



Effects of mode change in reinterviews. Bushery et al. (1992) and Royce (1994) exanuned the 
impact of mode for the original interviews and reinterviews for Round 2 and found that the mail- 
mail mode (mail return of questionnaires in both survey and reinterview) showed lower response 
variance than the telephone-telephone interview mode. Bushery et al. (1992) suggest that this 
finding may have resulted primarily from two factors: 

■ Only respondents who answered the original survey by mail were eligible for the mail 
reinterview. These respondents were likely to be more cooperative and answer the questions 
more carefully; and 
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ffl Respondents interviewed by mail may take time to look up the answers to questions from 
records or they may go through a more careful, lengthy thought process to provide the 
needed facts. Respondents interviewed by telephone may not feel free to take the time to 
look up records while the interviewer is waiting on the phone. 

Another possible explanation is that some mail respondents saved copies of their questionnaires 
and used the copies to complete their mail reinterview questionnaires. However, a school-by- 
school comparison of individual responses showed that no more than 6 percent of schools would 
have done this. 

2.6.4 Nonresponse 

The School Survey is subject to total school (or unit) and item nonresponse: 

o School nonresponse occurs when a questionnaire of acceptable quality is not obtained from 
an eligible school; and 

■ Item nonresponse occurs when entries are missing for one or more items on a questionnaire. 

At this time, there is no direct evidence on the magnitude of biases caused by these two kinds of 
nonresponse. However, information about the levels of school nonresponse for different 
subgroups of the survey population and the levels of item nonresponse for different questionnaire 
items provides some indication of the potential for nonresponse bias (Smith, Moonesinghe, arid 
Gruber, 1993; Scheuren et al., 1996; Monaco, Salvucci, Zhang, Hu, and Gruber, 1998). 

School nonresponse. For public schools, a few LEAs in Rounds 1 and 2 of SASS have refused 
participation completely, that is, they declined to complete the LEA questionnaire for the TDSS 
and they specifically requested NCES not to ask schools in their district to participate. In Round 
1, 35 school districts with 63 sample schools initially refused to have their schools participate in 
SASS. After contacts by U.S. Census Bureau representatives, 17 of these districts with 24 
sample schools reconsidered their positions and agreed to allow the U.S. Census Bureau to mail 
questionnaires to individual schools in their districts (Nash, 1988). Thus, the ultimate loss of 
schools at this stage was less than 0.5 percent of the public school sample. Although only a few 
schools and school districts were lost at this stage in Rounds 1 and 2, a few large districts were 
lost m each round, with adverse consequences for the quality of the data for the states in which 
those districts were located. For Round 3, refusals from LEAs were rare and the loss of schools 
at this stage was close to none. 

Weighted school response rates. Table 2-13 shows the weighted response rates for sample 
schools that were eligible for the school survey. The weighted response rate is the ratio of the 
sum of the basic weights for eligible responding schools to the sum of the basic weights for all 
sampled schools that are eligible for the survey, where a school's basic weight is the inverse of 
its selection probability. A response rate, weighted by the basic weight, provides a better 
indicator of the potential effect that nonresponse could have on survey estimates than does an 
unweighted rate. 



Table 2-13.— Weighted response rates for the School Survey, by sector: Rounds 1-3 



School sector 




Weighted response rate (percent)* 




Round 1 
(1987-88) 


Round 2 
(1990-91) 


Round 3 
(1993-94) 


Total 


88 


92 


90 


Public 


92 


95 


92 


BIA 


1: 


96 


99 


Private 


79 


84 


83 



*Not applicable. 

*Basic weight is used to reflect the probability of selection, 
SOURCES: NCES (1991a); Gruber et al. (1993, 1996). 



The overall school weighted response rate was 88 percent in Round 1, 92 percent in Round 2, 
and 90 percent in Round 3. Private schools had response rates that were around 10 percent lower 
than public schools in all three rounds. BIA schools had relatively high response rates. 

Table 2-14 shows the weighted response rates for public schools by state for Rounds 1 to 3. In 
Round 1, 27 states had weighted response rates close to 95 percent or better and 3 states (District 
of Columbia, Hawaii, and Maryland) had rates below 80 percent. In Round 2, 37 states had rates 
close to 95 percent or better and none had rates below 80 percent. The overall response rate fell 
slightly from Round 2 to Round 3, and only 13 states had rates of 95 percent or better. Maryland 
had the lowest response rate in both Rounds 2 and 3 (81 and 85 percent, respectively). 

For private schools, table 2-15 shows the weighted response rates by list and area frames and by 
association group and school type. For each round, the response rate of schools from the area 
sample was much lower (around 70 percent) than that of schools from the list sample. The 
response rates also varied among schools from the various association groups and across rounds 
within association groups. At least in part this is a reflection of the small samples on which 
many of the rates are based (see table 2-5). 

Analytic studies were conducted for Rounds 2 and 3 to examine variations in response rates for 
different categories of school. Some results are displayed in table 2-16. The study by Scheuren 
et al. (1996), which investigated variations in response rates for the first four characteristics in 
the table for Round 2, used a Bonferroni multiple comparison test with a 10 percent significance 
level for each characteristic (Ahmed, 1991). In the table S denotes a significant difference in 
response rates between at least one pair of categories for the characteristic; NS denotes no 
significant differences. Some significant category differences were found by region, school size, 
and urbanicity for public, and by school level and school size for private schools. However, as 
the table shows, the response rate differences are not great, particularly in the case of public 
schools. 
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Table 2-14. — Weighted response rates for public schools in the School Survey, by state: 
Rounds 1-3 



State - 

Total 

Alabama 

Alaska 

Arizona 

Arkansas 

California 

Colorado 

Connecticut 

Delaware 

District of Columbia 
Florida 

Georgia 

Hawaii 

Idaho 

Illinois 

Indiana 

Iowa 

Kansas 

Kentucky 

Louisiana 

Maine 

Maryland 

Massachusetts 

Michigan 

Minnesota 

Mississippi 

Missouri 

Montana 

Nebraska 

Nevada 

New Hampshire 

New Jersey 
New Mexico 
New York 
North Carolina 
North Dakota 

Ohio 

Oklahoma 
Oregon 
Pennsylvania 
Rhode Island 

South Carolina 
South Dakota 
Tennessee 
Texas 
Utah 

Vermont 
Virginia 
Washington 
West Virginia 
Wisconsin 

Wyoming 



Weighted response rate (percent)* 

Round I Round 2 Round 3 

(1987-88) (1990-91) (1993-94) 



92 


95 


92 


97 


96 


95 


97 


92 


88 


97 


95 


92 


95 


98 


94 


88 


95 


88 


99 


96 


92 


89 


93 


93 


91 


93 


88 


68 


86 


86 


98 


94 


95 


95 


97 


94 


78 


99 


92 


98 


99 


92 


95 


99 


94 


97 


100 


94 


96 


97 


96 


93 


98 


93 


90 


98 


92 


89 


94 


90 


97 


95 


92 


75 


81 


85 


95 


91 


94 


98 


97 


97 


91 


97 


95 


97 


97 


94 


85 


98 


95 


95 


98 


92 


96 


99 


89 


96 


96 


88 


97 


96 


98 


92 


88 


87 


88 


96 


93 


85 


88 


89 


91 


93 


90 


100 


98 


96 


95 


97 


93 


90 


96 


95 


97 


95 


93 


87 


96 


89 


99 


97 


90 


88 


97 


87 


95 


99 


96 


92 


98 


95 


87 


97 


94 


100 


98 


98 


99 


99 


93 


90 


92 


89 


100 


93 


96 


94 


98 


93 


94 


95 


94 


94 


98 


95 



♦Basic weight is used to reflect the probability of selection. 
SOURCES: NCES (1991a); Gruber etal. (1993, 1996). 
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Table 2-15. — ^Weighted response rates for private schools 
group and school type: Rounds 1-3 


in the School Survey, by association 




Weighted response rate (percent)* 


Round 1 


Round 2 


Round 3 


Association group (1987-88) 


(1990-91) 


(1993-94) 


Total 


79 


84 


83 


Schools from area sample 


67 


74 


71 


Schools from list sample 


— 


87 


84 


List sample by association group: 


Association of Military Colleges and Schools 


86 


91 


96 


Friends Council on Education 

National Catholic Education Association, Jesuit 


83 


91 


80 


Secondary Education Association 


90 


91 


89 


National Association of Episcopal Schools 


82 


89 


78 


National Society of Hebrew Day Schools 1 


1 


71 


79 


Solomon Schechter Day Schools | 


r 


85 


88 


Other Jewish J 


1 


70 


72 


Lutheran Church — Missouri Synod 1 


1 


96 


91 


Evangelical Lutheran Church — ^Wisconsin Synod 1 


^ 90 


98 


90 


Evangelical Lutheran Church in America J 


1 


96 


95 


Other Lutheran 




94 


88 


General Council of Seventh-day Adventists 


89 


94 


89 


Christian Schools International 
American Association of Christian Schools 


95 


94 


69 


International 

National Association of Private Schools for 


56 


59 


73 


Exceptional Children 


84 


87 


89 


American Montessori Society Schools, other Montessori schools 


82 


86 


83 


National Association of Independent Schools 


74 


85 


82 


National Independent Private School Association 


t 


t 


80 


Other private schools 


71 


81 


79 


List sample by school type: 


Catholic 


— 


91 


89 


Parochial 


— 


90 


88 


Diocesan 


— 


92 


91 


Private order 


— 


94 


88 


Other religions 


— 


80 


76 


Conservative Christian 


— 


74 


77 


Affiliated 


— 


88 


77 


Unaffiliated 


— 


77 


80 


Nonsectarian 


— 


82 


86 


Regular 


— 


77 


86 


Special emphasis 


— 


83 


81 


Special education 


— 


92 


93 



“Not available. 

*Not applicable. (Not listed as a separate association group.) 

*Basic weight is used to reflect the probability of selection. 

SOURCES: NCES (1991a); Gruber et al. (1993, 1996); McLaughlin and Boughman (1997); McLaughlin, O’Donnell, Ries, and Boughman (1995). 



Table 2-16. — ^Weighted response rates for public and private schools in the School Survey, by 
selected characteristics: Rounds 2 and 3 



Characteristics 




Weighted response rate (percent)* 






Round 2 
(1990-91) 


Round 3 
(1993-94) 


Public 


Private 


Public 


Private 


Region: 










West 


95 


84 


91 


78 


South 


95 


80 


92 


82 


Northeast 


92 


85 


90 


82 


Midwest 


98 


86 


94 


89 




S^ 


NS^ 


P=0.003 


PcO.OOl 


Urbanicity: 










Central city 


93 


83 


90 


83 


Urban fringe/large town 


94 


87 


91 


84 


Rural 


98 


82 


94 


84 




S^ 


NS^ 


PcO.OOl 


P=0.747 


School level: 










Combined 


94 


76 


90 


75 


Secondary 


96 


90 


93 


87 


Elementary 


95 


88 


92 


87 




NS^ 


S* 


P=0.010 


PcO.OOl 


School size: 










1-149 


97 


81 


95 


81 


150-499 


96 


88 


93 


86 


500-749 


95 


80 


92 


85 


750+ 


93 


87 


89 


83 




S^ 


S' 


PcO.OOl 


P=0.011 


School type: 










Nonregular 


* 


* 


90 




Regular 






92 










P=0.072 




Minority enrollment: 










<5.5% 


* 




94 




5.5-20.5% 






92 


t 


20.5-50.5% 






91 




>50.5% 


t 




90 


t 








P=0.004 




School sampled with certainty: 










Yes 






92 




No 






92 










P=0.514 




School sampled in 1990-91: 










Yes 






93 


88 


No 






92 


82 








P=0.606 


PcO.OOl 


Submitted a teacher list: 










Yes 


* 




93 


t 


No 


* 




61 


t 








PcO.OOl 




1991-92 PSS status: 










Respondent 


* 






85 


Nonrespondent 


* 






44 


Not in 1991-92 PSS 








63 










PcO.OOl 



^Not applicable. 

‘Basic weight is used to reflect the probability of selection. 

indicates a significant difference between the response rates for at least some of the levels of the variable at the 10 percent significance level. 
NS indicates that none of the differences are significant at that level. 

SOURCES: Scheuren et al. (1996); Mcxiaco et al, (1998). 
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Monaco et al. (1998) performed similar analyses for the 1993-94 Round, including the four 
characteristics examined in the 1990-91 study and adding the characteristics displayed in the 
lower part of table 2-16 (some of which related only to public schools, others only to private 
schools). They used an overall test of differences across categories; the significance level 
reached is indicated by the P-value in the table. For the four characteristics common to both 
analyses, the patterns of response rates are similar across rounds: response rates are highest in 
the Midwest, lowest for combined schools, and highest for small public schools and public 
schools in rural areas. Two of the new variables exhibited marked differences in response rate 
by category: public schools that submitted the teacher listing form had a substantially higher 
response rate (93 percent) than schools that did not (61 percent); and private schools that 
responded to the 1991-92 PSS had a much higher response rate (85 percent) than those that did 
not (44 percent). 

Both studies extended their analyses to develop logistic regression models predicting response 
rates using the characteristics studied as explanatory variables. Details are presented in the 
reports. 

Item nonresponse. Table 2-17 shows the data from published summaries of unweighted item 
response rates for the School Survey for the first three rounds (NCES, 1991a; Gruber, et al., 1993 
and 1996.) The rates refer to the status of each item after edits but prior to imputation, and the 
base for each rate is the number of questionnaires for which the item should have been answered. 



Table 2-17. — ^Percent of items with selected response rates for the School Survey, by sector: 
Rounds 1-3 



Sector and round 


Percent of items with 
response rates: 

> 90 percent < 75 percent 


Minimum of item 
response 
rates (percent) 


Round 1 (1987-88) 


Public 


64 


11 


43 


Private 


56 


8 


11 


Round 2 (1990-91) 
Public 


77 


1 


56 


Private 


77 


5 


67 


BIA 


87 


4 


60 


Round 3 (1993-94) 


Public 


83 


0 


83. 


Private 


77 


3 


61 


BIA 


84 


1 


70 



SOURCES: NCES (1991a); Gmber et al. (1993, 1996). 
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Item response rates tended to :be lower for items requiring respondents to report numerical 
amounts than for those requiring a choice among two or more categories. A common problem in 
Rounds 1 and 2 was a failure to check boxes for "none" when that was called for. Item 
nonresponse, problems in Round 1 and, subsequently, in Round 2 led to several changes in the 
content and format of the questionnaire. Because of these and other changes, the results for the 
three rounds shown in table 2-17 are not directly comparable. 

One item that caused particular problems in Round 1, for both public and private schools, was an 
item on staffing patterns that appeared in the form of a 3 x 28 grid. The 3 columns asked for 
number of teachers as on October 1, 1986; number of those no longer teaching on October 1, 
1987; and number of teachers in the category on October 1, 1987. In the 28 rows, teachers were 
to be classified by 27 different primary fields of assignment, with a total in the final row. This 
item appeared to be difficult for respondents to complete, as indicated by a combination of 
missing and inconsistent entries. NCES decided that the quality of data from this item was 
unacceptable and did not include the data in either its public or restricted use microdata files for 
the School Survey. In the Round 2 School Survey, some parts of this item were dropped; other 
parts were retained but were asked in a different format. 

Other items with high item nonresponse rates in Round 1 for both sectors included an item on 
availability of instruction and size of enrollment in "grades 13 and 14" (covering vocational and 
other "postgraduate" secondary education) and an item asking for a breakdown of prior year staff 
roles of teachers who were no longer in the profession in the current year. In Round 2, the 
grades 13 and 14 categories for enrollment were replaced by a single "postsecondary" category. 
The other item was retained but the number of separate response categories was substantially 
reduced. 

In Round 2, items with response rates below 75 percent included those relating to counts of part- 
time staff and to degree of difficulty in filling vacancies in selected categories. Problems with 
the items on part-time staff are attributed partly to the format of the item, which covered both full 
and part-time staff, and partly to respondents' uncertainty about the definition of "part time," 
especially in the smaller schools. 

In Round 3, none of the items on the public school questionnaire had response rates below 75 
percent. For the private school questionnaire, the block of items (item 31c) that had response 
rates below this level asked about teacher vacancies. The items asked whether there were 
vacancies for the school year, and, if so, about the methods used to cover them and about the 
difficulty or ease of filling vacancies in each of the 14 fields. 

2.7 DATA PROCESSING AND ESTIMATION 

This section summarizes the sequence of processing operations for the School Survey that 
occurred between the receipt of questionnaires in the U.S. Census Bureau's processing facility in 
Jeffersonville, Indiana, and the production of a clean data file of schools. The processing units at 
Jeffersonville handled the activities with large clerical elements and transmitted the data files 
electronically to U.S. Census Bureau's headquarters in Suitland, Maryland, which handled most 
of the computerized operations. The activities discussed in this section include clerical review of 
questionnaires and data entry operations (2.7.1); computer edits and file preparation (2.7.2); 
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imputation of missing data (2.7.3); construction of weights (2.7.4); and methods of variance 
estimation with SASS data. 

2.7.1 Clerical review and data entry of hard-copy questionnaires 

Upon receipt of the hard-copy questionnaires, clerks at Jeffersonville assigned codes to each 
questionnaire to indicate its interview status (e.g., complete interview, noninterview, or out of 
scope). Then they performed a general clerical edit that included reviewing all entries for 
legibility and consistency (e.g., changing "one" to "1" and rounding fractions to integers). For 
CATI questionnaires, these edits were built into the data collection programs. 

After clerical review edits, the questionnaires were sorted by school type (public, private, and 
BIA) and by interview status. The specification for data entry called for 100 percent independent 
verification of all data keyed from the questionnaires. All errors identified during the keying 
operation were corrected, with problem cases referred to supervisors. 

Error resolution. Attempts were made to resolve some critical errors identified during clerical 
edit and also the following computer pre-edit operations (see section 2.7.2) by recontacting 
respondents by telephone. In Rounds 2 and 3, telephone callbacks were used, in particular, to 
resolve large differences between reported enrollments and teacher counts and the expected 
values of these items based on the CGD file used for sampling schools. In Round 1, many 
discrepancies of this kind were detected only when these items were compared at the aggregate 
level after data processing was completed (see section 2.8.1), and their resolution at that stage 
caused significant delays in the production of clean data files. Starting with Round 2, the CCD 
values for these items were included on the mailing labels for the school questionnaires, so that 
large discrepancies (differences of over 35 percent) could be detected and resolved by the 
clerical reviews. Any discrepancies not resolved were flagged for resolution during computer 
pre-edit. 

Checks for invalid entries for specific items, inconsistencies between items and other problems 
were included in both the computer pre-edit and edit operations. In the pre-edit, a listing of 
rejected schools and items was produced and sent to Jeffersonville, where the clerical staff 
reviewed the listings in conjunction with the questionnaires and, as needed, recontacted 
respondents. In the edit, problems detected in the corrected data files were resolved through 
programmed instructions to blank or impute problem items; there were no attempts to contact 
respondents at this stage. Some questionnaires were rejected in the edit and the schools were 
treated as nonrespondents if values were still missing or out of range for selected key items. 
Round 1 used fairly stringent criteria in pre-edit check and extensive followup to collect data 
from schools on key items. The pre-edit criteria was relaxed for subsequent rounds and there 
were less extensive followup with schools. This change of procedure may affect the percent of 
schools treated as nonrespondents at this stage. 

Field ojfice edit in Round 2. A field office edit check on the quality of the data collected by field 
representatives during the telephone followup operations was conducted in Round 2. Part of this 
edit required the clerk to compare the numbers of students and teachers reported in the school 
questionnaire with the numbers of students and teachers from the CCD shown on the label for 
that questionnaire. If the CCD count was twice (or half) the number reported on the 
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questionnaire, the field representative was required to provide a written explanation of the 
difference. 

2.7.2 Computer edits and file preparation 

The computer pre-edit was designed to identify inconsistencies and invalid entries for key data 
items. This pre-edit step generated a list of cases with problems and displayed the items rejected 
by the pre-edit program along with the error messages describing the problems. The listings of 
rejected schools and items were reviewed in conjunction with the questionnaires, and, as needed, 
respondents were recontacted by telephone to clarify their responses. Entries confirmed to be 
correct were checked for acceptability; those that were erroneous but could not be corrected were 
deleted; and corrections made were marked on both the rejection list and the original 
questionnaire. 

Gruber et al. (1996, table VII-2) list the reasons for pre-edit rejections for public school data and 
the number of records rejected as a result of these edits. School records were rejected when there 
were large discrepancies between data on the frame and data reported by schools. For example, 
over 900 records were rejected because the numbers of teachers reported by the schools were 
different from the frame numbers by 25 percent or more. A small number of schools were 
rejected because they reported no teachers or no students and, thus, were ineligible for the 
School Survey. The vast majority of the problem cases were found in Montana, New Jersey, 
North Dakota, South Dakota, Wyoming, and the District of Columbia. The corrections made at 
this stage were incorporated into the data files. 

Computer edits consisted of the following components: a range edit with out-of-range entries 
being deleted; a consistency edit with inconsistent entries either being made consistent or 
deleted; and a blanking edit with inappropriate entries being deleted and entries that were blank 
but should have been answered being assigned a "not answered" code. To some extent, the 
computer editing repeated a number of the checks performed at the pre-edit stage, such as 
checking for invalid entries or inconsistencies. However, all errors detected at this later stage 
were resolved through programmed instructions, without further reference to either the 
respondent or the questionnaire. 

This stage of editing checked for inconsistencies both within and between items and made use of 
available information to fill incomplete or omitted responses. For example, in Round 3, if a 
school's total enrollment was not reported in item 8 of a public school record but enrollment by 
grade was reported in item 7, the omitted response in item 8 was replaced by the sum of the 
enrollments from item 7. Gruber et al. (1996, figures VII-5-7) give the number of changes made 
to each item in the Round 3 public, private, and BIA school questionnaires. Most school records 
had some edit changes made at this stage. 

Edit rejection rates. School records with missing or out-of-range data for selected key items 
after editing were rejected and treated as noninterviewed schools. The rules for determining a 
school's final response status (i.e., interviewed, noninterviewed, or out of scope) after computer 
edits are described in the SASS Data File User's Manual (Gruber et al., 1993 and 1996). In 
Round 3, for example, the change from interviewed to noninterviewed status affected less than 1 
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percent of schools that had initial interviewed status (300 public schools, 30 private schools, and 
1 BIA school). 

Assignment of locale codes in Round 2. Round 1 reinterviews (see section 2.6.3) and cognitive 
interviews using Round 2 pretest questionnaires showed that responses to the question, "Which 
best describes the community in which the school is located?" had moderate response variance. 
The same question was used in Round 2, but a separate locale or "urbanicity" code was 
developed by matching each school's mailing address to U.S. Census Bureau geographic files 
containing population density data, standard metropolitan statistical area codes, and urban/rural 
codes. The same locale codes were used for the school and school administrator data files. 
These more rigorously defined locale codes will sometimes differ from the Codes based on self- 
reports of community type (Kaufman and Huang, 1993, section 1.4.4; Johnson, 1993). 

2.7.3 Imputation 

Imputation is the procedure of assigning values for questionnaire items that should have been 
answered but were not or for which the answers were deleted in editing. Several alternative 
imputation methods are used in SASS; some make use of available data reported for the same 
school on the school questionnaire; some make use of data from other SASS components such as 
the School Principal Survey and the TDSS for public schools; and, if necessary, some assign 
valid values from a donor school with similar characteristics. Imputation is used to compensate 
for item nonresponse among interviewed schools. Weighting adjustment procedures (see section 
2.7.4) are used to compensate for noninterviewed schools. 

The SASS Data File User's Manuals for Rounds 2 and 3 (Gruber et al., 1993, 1996) provide 
details on the imputation rates and methods, and additional details about item-by-item 
specifications for imputation are available in SASS Specifications Memoranda covering each 
survey and each questionnaire. For Round 3, Gruber et al. (1996, figures VIII-14 to 22) 
document the imputation method for each item on the public and private school questionnaires. 
This section summarizes the extent of imputation and the three basic stages of imputation 
employed for the School Survey. 

Extent of imputation: All items on the School Survey questionnaires were imputed at each 
round. In Round 1 there were two items on the private school questionnaire that were not 
imputed initially, an item on place of operation and an item concerning staffing patterns. These 
items were imputed later when the Round 1 data was released with data from Rounds 2 and 3 on 
a CD-ROM (NCES, 1998). 

At several stages during data processing, some respondents' or interviewers' initial entries on the 
questionnaires were changed or deleted, or values (including 0) were supplied for items initially 
left blank on the questionnaire. Except when these changes were the result of followup contacts 
with respondents, the process of changing entries or of assigning values for deleted entries was 
part of imputation. 

Some imputation was done during the computer editing and a very limited amount in earlier 
stages of processing, including the initial clerical edit and the clerical resolution of pre-edit 
rejections. Most changes during the clerical operations resulted from followup contacts with 
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respondents; clericar imputation was permitted only in a few situations where the correct entry 
was obvious from other information on the questionnaire. Table 2-17 provides an indication of 
the levels of item nonresponse after corrections had been made at the pre-edit stage. 

Imputation methods. Most of the imputation for each round of the School Survey was done in a 
computerized imputation operation conducted in stages as follows: 

■ Stage 1 deductive imputation. Stage 1 imputation replaces the missing or inconsistent values 
with values that could be derived with a reasonable degree of assurance from other available 
data for the same school. The data for the same school were extracted from one of the 
following sources: other items on the school's questionnaire, items from other SASS 
components that pertained to the same school, and information from the sampling frame. 

The imputation for Round 3 also involved the use of ratio adjustments to .correct for some 
inconsistencies between items. For example, if the total of the numbers of students by 
race/ethnicity did not equal the total enrollment reported elsewhere in the questionnaire, the 
numbers of students by race/ethnicity were adjusted by applying the proportion of students in 
each race/ethnicity group to the total number of enrolled students, thus making the detailed 
numbers consistent with the total. The same procedure was used to adjust any discrepancies 
between the total teacher count, the numbers of teachers by race/ethnicity, and the numbers 
of teachers by part-time and full-time status. The adjustments were applied to both public 
and private schools. 

■ Stage 2 hot deck imputation. Data that remained missing after the stage 1 imputation were 
processed in the stage 2 sequential hot deck imputation procedure. This procedure sorted 
schools into imputation classes that contained schools with similar characteristics. The value 
assigned for a missing item for one school was then the value for that item in another school 
(the donor) in the same imputation class. The variables used to define imputation classes 
included metropolitan status, percent minority enrollment, and size of enrollment (see NCES, 
1992; Gruber et al., 1993, 1996). The choice of imputation classes was based on analyses of 
response propensities and characteristics of respondents and nonrespondents. Hot deck 
imputations were sometimes used on a transformed variable. For example, if a school 
reported that it had a remedial reading program but did not report the number of students 
served by the program, the school's total enrollment would be multiplied by the proportion of 
enrollment served in rernedial reading programs at another school with similar characteristics 
(Kaufman and Huang, 1993^ section 8.2). 

■ Stage 3 clerical imputation. Round 3 included a clerical imputation step for items not 
completely imputed by the first two stages. This method involved reviewing the data record, 
the sample file record and, in some cases, the school questionnaire before deriving an entry 
consistent with the information from the various sources. This procedure was used on 12 
items in the public school questionnaire and 13 items in the private school questionnaire. It 
was used in situations when there was no suitable record to use as a donor for hot deck 
imputation, when the computer imputation method produced an imputed entry that was 
outside the acceptable range for an item, or when there were very few cases where an item 
was unanswered (usually less than 10). 
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Imputation of items on the BIA school questionnaire was done clerically in both Rounds 2 and 
Round 3 because of the small number of BIA schools and the relatively low level of item 
nonresponse. The BIA school records were sorted by state and size so that records for similar 
schools were close together. The questionnaires were reviewed for notes and other entries that 
were useful in deciding the entries to be imputed. If an item cbuld not be completed based on the 
information on the questionnaire, entries from the record for a similar school were used. Gmber 
et al. (1996, figure VIII-23) give the imputation method and the percent of entries imputed by the 
method for each BIA questionnaire item. 

2.7.4 Weights 

The weighting procedures used in the School Survey have three purposes: to take account of the 
school's selection probabilities; to reduce biases that may result from unit nonresponse; and to 
make use of available information from external sources to improve the precision of sample 
estimates. 

The weighting procedures for public, private, and BIA are quite similar, with minor variations at 
some stages. For each sector in Rounds 1 and 2, the overall weights are the product of the 
following four factors: a basic weight, a sampling adjustment factor, a school nonresponse 
adjustment factor, and a first-stage adjustment factor to known frame totals. In Round 3, a fifth 
factor — a second-stage ratio adjustment factor — is added for private schools. 

Basic weight. This is the inverse of the selection probability for the school. Schools selected 
with certainty have a basic weight of 1. 

Sampling adjustment factor: This factor is used to take account of special circumstances that 
affected a school's probability of selection. The circumstances included frame duplications not 
discovered prior to sample selection, including the special case where a sample school was 
discovered to have merged with another one also included in the frame. Another instance is the 
one described earlier, in which some special education programs, with operations at several 
locations, were identified in the frame as single schools. In this situation, locations not already 
included in the frame as regular schools were subsampled. 

School nonresponse adjustment factor. This factor is applied to the interviewed schools for 
which acceptable questionnaires were obtained to compensate for sample losses due to 
noninterviewed schools; The factors, which are calculated separately for specified adjustment 
cells, are the ratios of the summed sampling weights (product of the first two factors above) for 
all eligible sample schools in the cell to the summed weights for those that responded. 

The nonresponse adjustment cells used by type of school for the three rounds of the School 
Survey were the following: 

■ For public schools, the cells were defined in sequential order by state, grade level 
(elementary, secondary, or combined), urbanicity, and enrollment size class. 

■ For BIA schools, the cells were defined by grade level, and in the case of elementary schools 
they are further defined by enrollment size. 
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■ For Native American schools, the cells were defined by grade level, state strata, and in the 
case of elementary schools also by enrollment size. 

■ For private schools, the Round 1 adjustment cells were defined by census region, affiliation 
(Catholic, other religious, and nonsectarian), grade level, and urbanicity for schools on the 
list frame; and census region, affiliation (same broad classes), and grade level for schools on 
the area frame. The Round 2 and Round 3 cells for schools on the list frame were based on 
the association groups in the estimation domain (18 groups in Round 2 and 19 groups in 
Round 3), grade level, and urbanicity (for Catholic and other private school groups only). 
The cells for schools on the area frame were based on affiliation (3 broad groups), grade 
level, and enrollment size class. 

Cells with small samples (less than 15 public and private schools or less than 10 BIA or Native 
American schools), and cells with high adjustment factor values (1.5 or larger for public or 
private schools, or 2.0 or larger for BIA or Native American schools), were combined following 
designated rules (Gruber et al., 1993, 1996). 

The nonresponse adjustment factors are based on the assumption that the probability of 
nonresponse may vary between cells but does not vary among individual schools within cells. 
Therefore, it is important to define the cells in a way that makes nonresponse probabilities as 
homogeneous as possible within cells. Shen, Parmer, and Tan. (1992) examined the correlates of 
nonresponse in Round 2 School Survey samples of public and private list sample schools, using a 
variety of analytic procedures. Their analyses supported the use of the Round 2 adjustment cells 
for public schools but suggested a change in the order of collapsing cells when collapsing is 
necessary. For the private school list frame, they suggested the use of enrollment size in creating 
adjustment cells, in addition to the variables used in Round 1, as well as a change in the order of 
collapsing cells. Their suggestions were adopted in part for the Round 3 School Survey, as 
indicated above. 

First-stage ratio adjustment factor. This adjustment is used to adjust for differences between 
expected and actual sample sizes. Like the nonresponse adjustment factors, these ratio 
adjustment factors are calculated separately for specified adjustment cells. The factor for each 
cell is the ratio of the total number of schools in the frame in that cell to the sample estimate of 
that number, based on all schools selected, without regard to their final response and eligibility 
status. Schools that turned out to be ineligible or did not respond had to be included in the 
denominator in order that the weighted sample total would conform to frame totals. 

For public schools and private schools in the list sample, the cell definitions and collapsing rules 
were similar, although not always identical, to those used for the nonresponse adjustments. For 
the private school area sample, the frame ratio adjustment factors use weighted area frame PSU 
totals as the control totals. For Round 2 this adjustment was applied only in the certainty PSUs 
since all schools in the area frame were included in the sample for the noncertainty PSUs. For 
Round 3, adjustment was not applied to the private schools in the area sample since, as described 
in section 2.4.2, the area frame schools in certainty PSUs were incorporated into the list frame 
and those in noncertainty PSUs were sampled with certainty. 
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Second-stage ratio adjustment factor for private schools in Round 3. Round 3 added a second- 
stage ratio adjustment factor to the private school sampling weights to adjust the SASS sample 
estimates to more up-to-date independent control counts from the 1994 PSS. Since Round 3 of 
SASS and the 1993-94 PSS were fielded in the same school year, there were up-to-data school 
counts to support this adjustment. Adjustment factors are derived using the ratio of the weighted 
1993-94 PSS estimates of schools to the weighted 1993-94 SASS sample estimate of schools 
within adjustment cells. This adjustment has the advantage of reducing sampling variability and 
maintaining better consistency with the PSS. Since this adjustment was not used in previous 
rounds, this change needs to be taken into account when examining trends across rounds. 

Analogous adjustments for public schools to up-to-date CCD counts have yielded unsatisfactory 
results due to definitional and other differences between CCD and SASS. Therefore, a second- 
stage ratio adjustment was not applied for public schools (Abramson et al., 1996). Scheuren and 
Li (1995, 1996) and Holt, Kaufman, Scheuren, and Smith (1994) have explored the use of 
modified generalized least squares methods to derive the sampling weights in order to achieve 
consistency between SASS and PSS estimates of schools, teachers, and students. However, this 
approach did not provide a satisfactory solution for attaining consistency on all three counts. 

2.7.5 Variance estimation 

Balanced half-sample replication. In Rounds 1 and 2, the balanced half-sample replication 
method is used to estimate the sampling errors associated with estimates for all surveys in SASS. 
Replicates are subsamples of the full sample. For the balanced half-sample procedure, each 
replicate or subsample consists of approximately one-half of the full sample of schools. Each 
sample school is included in one-half of the replicates, except for schools selected with certainty 
which are included in all replicates. The statistic of interest, such as the number of students at a 
specified grade level, is estimated from each replicate. Then, the mean square error of the 
replicate estimates around the full sample estimate provides an estimate of the variance of the 
statistic. 

A total of 48 replicates was designated for each of the SASS surveys in both rounds. For the 
School Survey, special procedures were used to ensure that the effect of controlling overlap of 
the Round 1 and Round 2 samples would be properly reflected in estimates of variance for 
changes occurring between the two rounds. In Round 1, the s^ue overall nonresponse and frame 
ratio adjustment factors were used for each replicate. In Round 2, these factors were calculated 
separately for each replicated Details on the procedures for designating the replicates are 
provided in the Data File User's Manuals (NCES, 1991a; Gruber et al., 1993) and the Sample 
Design and Estimation Report for these two rounds (Kaufman, 1991; Kaufman and 
Huang, 1993). 

Each public use data file contains 48 sets of replicate weights needed to produce balanced half- 
sample replicated variance estimates, so that file users can estimate the sampling errors for 
statistics that are of interest to them. The same procedures are used to estimate the sampling 
errors that are presented in all SASS publications (e.g.. Education Data Tabulations by Bobbitt, 
Broughman, and Gruber, 1995). 
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Bootstrap replication. Round 3 used a bootstrap method to create the 48 replicate samples in 
place of the balanced half-sample method, with the bootstrap replicate samples being selected 
without replacement. For private schools from the area frame, the balanced half-sample method 
was agmn used to create the replicate weights in the same way as previous rounds. Variance 
estimation in Round 3 can be performed with the same variance software packages used to 
perform the balanced half-sample method. 

A concern with the balanced half-sample replication method is that it treats the schools as 
sampled with replacement and hence ignores the finite population connection (fpc) factor. Since 
the School Survey employs high sampling fractions in a number of domains — particularly small 
states and association groups— ignoring the fpc's in variance estimation will lead to 
overestimates of the variances for some estimates. The bootstrap replication method is designed 
to avoid this overestimation (Kaufman, 1992,1993,1994). 

Broene and Rust (2000) compare three software packages for estimating standard errors in 
NCES surveys, including SASS, two of which employ a linearization approach and the third 
employs a replication approach. . They point out that since the replicate weights developed for the 
School Survey and other SASS surveys have been constructed in Round 3 with the aim of 
accounting for the fpc's, standard errors computed by replication methods should avoid 
overestimation. The linearization packages, when used without the inclusion of fpcs, tend to 
overestimate standard errors in SASS surveys. 

Table 2-18 shows, for public schools, estimates of the number of schools, teachers, and students 
by state from Round 3 of the School Survey, together with the CVs of those estimates (where the 
CV is the standard error of the estimate divided by the estimate). A commonly used standard for 
acceptable precision is that a CV should be less than 10 percent. All the CVs in table 2-18 are 
much lower: for the number of schools, Nebraska has the highest CV at 3.6 percent; for the 
numbers of teachers and students, the CVs are somewhat larger, but only one exceeds 6 percent 
(a CV of 6.8 percent for the number of students in Nebraska). 

Table 2-19 shows the corresponding estimates for private schools in Round 3 of the School 
Survey. The CVs vary markedly across the different types of private schools in the table, from 
under 1 percent to just over 10 percent (for the numbers of teachers and students in nonsectarian 
special emphasis schools). The majority of the estimates have CVs of less than 7 percent. 

Generalized variance Junctions and design effects. Generalized variance functions (GVFs) 
provide a simple means for obtaining approximate sampling errors associated with survey 
estimates. They are useful for analysts who do not work with microdata files or lack the 
software for computing sampling errors for complex sample designs. Salvucci and Holt (1992) 
confirmed the feasibility of including GVFs in SASS publications. Specific parameter values for 
the GVFs have been computed for Round 1 by Salvucci, Holt, and Moonesinghe (1994), and 
have been used in internal analyses at NCES. The parameter values for the GVFs and the design 
effects of estimates were again computed for Round 2 (Salvucci, Weng and Kaufman, 1995). 
These estimates have not been updated for Round 3. 
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Table 2-18. — ^Number of public schools, teachers, and students, and coefficients of variation 
(CV), by state: Round 3 



State 


Number of 
schools 


CV 

(percent) 


Number of 
teachers 


CV 

(percent) 


Number of 
students 


CV 

(percent) 


Total 


■ 80.740 


0.2 


2.561.2944 


0.8 


41.621.660 


0.9 


Alabama 


1,274 


1.0 


44,791 


3.5 


745,963 


3.5 


Alaska 


478 


1.2 


8,152 


4.7 


127,130 


4.8 


Arizona 


1,057 


0.8 


37,600 


3.2 


685,519 


3.1 


Arkansas 


1,084 


0.8 


30,621 


3.1 


460,286 


4.1 


California 


7,319 


1.0 


209,032 


3.6 


4,804,574 


3.6 


Colorado 


1,329 


1.1 


35,723 


5.2 


616,434 


4.9 


Connecticut 


964 


1.0 


35,465 


2.9 


472,718 


2.6 


Delaware 


169 


1.2 


7,027 


3.8 


107,701 


3.7 


District of Columbia 


160 


2.8 


5,185 


3.3 


75,948 


3.9 


Florida 


2,348 


1.2 


106,535 


3.0 


1,888,762 


3.1 


Georgia 


1,723 


0.6 


74,907 


1.9 


1,194,072 


2.1 


Hawaii 


234 


0.5 


11,137 


4.1 


173,041 


4.7 


Idaho 


573 


1.0 


12,166 


4.4 


218,179 


4.6 


Illinois 


3,884 


1.6 


111,511 


2.5 


1,747,678 


2.6 


Indiana 


1,869 


0.4 


57,32 


2.8 


972,991 


2.8 


Iowa 


1,518 


1.3 


35,861 


4.1 


484,443 


4.1 


Kansas 


1,450 


0.6 


31,164 


3.9 


431,981 


4.5 


Kentucky 


1,327 


2.1 


41,571 


4.0 


693,316 


4.3 


Louisiana 


1,446 


0.5 


48,948 


2.2 


791,318 


2.3 


Maine 


721 


1.0 


15,658 


4.3 


207,975 


4.7 


Maryland 


1,185 


0.6 


43,862 


1.6 


753,706 


1.6 


Massachusetts 


1,689 


1.1 


58,416 


2.8 


776,415 


2.8 


Michigan 


3,159 


1.2 


83,288 


4.0 


1,491,699 


4.3 


Minnespta 


1,492 


2.1 


44,150 


4.6 


705,021 


5.1 


Mississippi 


957 


0.8 


29,851 


3.9 


531,874 


3.8 


Missouri 


2,082 


0.5 


62,454 


3.6 


938,836 


4.2 


Montana 


890 


0.6 


12,851 


4.6 


175,611 


5.3 


Nebraska 


1,296 


3.6 


20,411 


5.5 


248,016 


6.8 


Nevada 


365 


1.6 


12,822 


3.4 


231,088 


3.3 


New Hampshire 


445 


0.0 


12,299 


5.9 


174,563 


6.1 


New Jersey 


2,195 


1.4 


83,935 


4.0 


1,097,841 


5.0 


New Mexico 


663 


0.5 


19,265 


3.9 


323,001 


4.0 


New York 


3,904 


0.7 


178,701 


3.4 


2,593,562 


3.7 


North Carolina 


1,927 


1.5 


72,305 


3.8 


1,090,802 


3.8 


North Dakota 


582 


1.8 


8,404 


5.3 


1 15,635 


6.0 


Ohio 


3,636 


1.7 


111,518 


3.2 


1,816,266 


3.4 


Oklahoma 


1,763 


1.0 


42,220 


3.5 


579,583 


4.0 


Oregon 


1,184 


0.6 


25,706 


3.6 


478,877 


4.4 


Pennsylvania 


3,128 


1.4 


114,571 


5.3 


1,805,243 


5.6 


Rhode Island 


, 295, 


. 2.4 


9,217 


5.9 


124,230 


4.9 


South Carolina 


1,081 


0.9 


39,623 


4.4 


630,309 


5.0 


South Dakota 


661 


2.0 


10,579 


4.3 


139,525 


5.3 


Tennessee 


1,522 


0.7 


47,662 


3.0 


840,505 


3.8 


Texas 


5,890 


0.9 


223,800 


4.0 


3,342,778 


4.4 


Utah 


674 


1.6 


19,884 


3.4 


454,114 


3.6 


Vermont 


318 


2.2 


7,327 


4.8 


91,787 


5.1 


Virginia 


1,698 


2.4 


64,937 


4.9 


958,091 


4.8 


Washington 


1,806 


0.6 


48,452 


3.7 


913,048 


3.4 


West Virginia 


898 


3.3 


21,473 


5.2 


316,190 


5.3 


Wisconsin 


2,014 


0.9 


62,958 


3.3 


880,935 


3.2 


Wyoming 


411 


2.0 


7,567 


5.0 


102,484 


5.4 



SOURCE: CVs were computed using estimates and standard errors from Bobbitt, Broughman, and Gruber (1995). 
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Table 2-19. — ^Number of private schools, teachers, and students, and coefficients of variation 
(CV), by state: Round 3 



Private school type 


Number of 
schools 


CV 

(percent) 


Number of 
teachers 


CV 

(percent) 


Number of 
students 


CV 

(percent) 


All private schools 


26,093 


0 


378,365 


1.5 


4,970,548 


1.2 


Catholic 


8,351 


0 


149,840 


1.0 


2,516,028 


0.6 


Parochial 


5,109 


2.2 


77,382 


2.7 


1,386,668 


2.5 


Diocesan 


2,436 


4.1 


46,941 


3.7 


791,605 


3.8 


Private order 


806 


6.5 


25,516 


5.9 


337,756 


6.1 


Other religious 


12,180 


1.7 


140,279 


2.6 


1,686,069 


2.9 


Conservative Christian 


4,664 


3.8 


49,676 


5.5 


641,828 


5.5 


Affiliated 


3,437 


4.8 


52,395 


4.7 


580,666 


4.2 


Unaffiliated 


4,079 


6.2 


38,209 


6.4 


463,575 


7.3 


Nonsectarian 


5,563 


3.6 


88,246 


4.3 


768,451 


4.8 


Regular program 


2,484 


7.3 


57,306 


5.5 


539,785 


6.4 


Special emphasis 


1,788 


7.5 


15,955 


11.7 


141,929 


10.6 


Special education 


1,290 


9.2 


14,985 


8.9 


86,738 


8.7 



SOURCE: CVs were computed using estimates and standard errors from Bobbitt, Broughman, and Gruber (1995). 



2.8 EVALUATION OF ESTIMATES 

This section describes comparisons of weighted School Survey estimates with data from other 
sources, including other SASS surveys, the CCD, the PSS and, to a limited extent, data from 
other agencies and organizations. Comparisons of survey estimates with other data can be made 
both prior to and following publication. In Round 1, as soon as weighted data files were 
available, state-level estimates of numbers of schools, total enrollment, and teacher counts were 
compared with CCD and QED data. The large differences in estimates that were found for 
public schools in some states led to a substantial amount of review of individual school records 
and questionnaires and, in some instances, recontacts with schools in order to correct erroneous 
data. A similar set of operations was undertaken in Round 2, except that QED data were not 
used in the comparisons. The prepublication review and correction operations are referred to as 
the "prepublication review." Comparisons of the final estimates with data from other sources is 
termed the "evaluation of published estimates." The following discussion includes the 
prepublication review and ev^uation of estimates for Round 1 (2.8.1) and Round 2 (2.8.2) and 
the evaluation of estimates in Round 3 (2.8.3). 

2.8.1 Prepublication review and evaluation, Round 1 

Total student enrollment. An initial comparison of survey estimates and CCD counts of total 
enrollment by state identified several states, predominantly in the Midwest, for which public 



school enrollment estimated from the School Survey was much higher than expected. To 
identify the individual schools that might account for the differences, a list was prepared of 972 
public schools whose reported enrollment exceeded the expected value, based on the QED 
frame, by 35 percent or more. Questionnaires for 687 of these schools (excluding those in the 
states with the lowest ratios of reported to expected enrollment) were subjected to detailed 
reviews. About one-fourth of the 687 questionnaires were accepted as correct. About one-half 
of them had been partially completed for the school district or for more than one school and there 
was enough information on the questionnaire to make corrections. For the remaining one-fourth, 
it appeared that all questionnaire items had been completed for a school district, two schools, or 
the wrong school. These cases were assigned to NCES and U.S. Census Bureau staff for 
telephone reinterviews, which were successfully completed for about four-fifths of them. 

This experience led to the following conclusion; 

Our review of the questionnaires and phone conversations with school 
secretaries and principals lead us to believe that these errors were made 
because the respondents misread the first question on the school 
questionnaire, or because of their employment positions (district 
superintendent, assistant superintendent, principal for two schools, etc.), 
they assumed we wanted information for all schools under their 
jurisdiction rather than the school named on the questionnaire label. 

(Fondelier, 1989a). 

Number of teachers. After the corrections based on the above review had been incorporated in 
the School Survey data file, further comparisons with CCD data showed that estimates of the 
number of public school teachers for some states were still much higher than expected, based on 
the CCD. Program specifications were reviewed for possible errors although none were 
detected. 

A list of suspect schools was compiled, based on several criteria, such as student/teacher ratios, 
comparison of head counts and FTE counts of teachers, and comparison of FTE counts for the 
school and for the district in which it was located. Two of the criteria used for flagging schools 
were met by more than one-eighth of all sample schools. 

The review of suspect’ schools was based primarily on examination of computer listings. There 
were no recontacts with schools and only a few of the original questionnaires were examined. 
The review led to corrections for 281 school records. The conclusion concerning sources of the 
problem was: 

The problem of FTE teacher overestimates was caused chiefly by the 

respondents' reporting district enrollment for some schools he 

inconsistency between the district enrollment and the school teacher 
count triggered the edit procedure which increased the teacher count 
[which had not been subsequently returned to its original value when the 
enrollment count was corrected]. Other causes were probably poorly 
recorded entries which were misread by the keyers and a lack of 
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understanding by some respondents of "full-time equivalent". (Fondelier, 

1989b). 

The review found that 13.7 percent of the interviewed schools with one or more part-time 
teachers had identical entries for the head count and FTE number of teachers. However, no 
changes were made for these schools; it was believed that "... changing the FTE entries for these 
cases would not significantly alter the weighted teacher counts for the states in which they were 
located" (Fondelier, 1989b). 

Round 1 evaluation of published estimates. Information obtained from prepublication reviews, 
comparisons with CCD data, and other sources of information within NCES leads to the 
following conclusions about the quality of final estimates from the School Survey in Round 1: 

■ School counts estimated from SASS were lower than those obtained from NCES's Public 
Elementary/Secondary School Universe Survey, a part of the CCD program. At the national 
level, CCD counts exceeded SASS estimates by 6 percent, with the differences being greatest 
for Nebraska — 44 percent. North Dakota — 42 percent. South Dakota — 38 percent, and 
Montana — 15 percent. These differences were due in part to definitional differences between 
the QED (frame for the Round 1 School Survey), which defines schools in terms of physical 
locations, and the CCD, which defines them in terms of administrative units (NCES, 1991a). 

■ In Nebraska, the QED was found to have excluded some small (elementary grades only) 
LEAs with a total of about 275 schools and 2,800 students. The schools, students, and 
teachers in these LEAs were not included in SASS (Hammer and Gerald, 1991, p.22). 

■ The FTE teacher counts from the School Survey are likely to be overestimates. In the 
average state, 19 percent of the schools having part-time teachers reported identical FTE and 
head counts for teachers (NCES, 1991a). 

■ FTE teacher counts for Hawaii from SASS were substantially higher than the CCD counts. 
For the latter, the state had reported in terms of "authorized" positions that were filled, 
whereas in SASS the schools correctly reported all teachers, regardless of whether their 
positions were officially authorized (Fondelier, 1989b). 

Smith and Salvucci (1989) compared preliminary estimates of private school enrollment from the 
Round 1 SchooT Survey with estimates from the October Education Supplement to the U.S. 
Census Bureau's Current Population Survey (CPS) and, for Catholic schools, compared SASS 
estimates of school counts and enrollment with data available annually from the National 
Catholic Educational Association (NCEA). The CPS estimate of enrollment in private 
elementary and secondary schools in October 1987 was 4,420,000. This estimate was 16.5 
percent below the preliminary SASS estimate of 5,291,000 and 15.3 percent below the final 
SASS estimate of 5,218,000. CPS estimates of private school enrollment were also significantly 
below estimates from NCES sources other than SASS for 1983, 1985, and 1988. The report 
asserted that "... these differences cannot be fully explained without a major benchmarking 
study." 
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The NCEA figures for Catholic schools are based on an annual census covering all . schools 
identifiable as Catholic, including those operated by private boards of control and not affiliated 
with a parish or diocese. 

As shown in table 2-20, the SASS final school and enrollment estimates exceeded those of the 
NCEA by 5.9 and 7.6 percent, respectively. For the following school year, the estimate of 
Catholic school enrollment from NCES's Early Estimates Survey exceeded the NCEA's count by 
9.2 percent. A subsequent review of school lists available annually from the Council for 
American Private Education (1992) suggested that the higher estimates from SASS may be 
accounted for in part by schools that are affiliated with the U.S. Catholic Conference but were 
not included in the NCEA annual census. 

Table 2-20. — ^Enrollment in private Catholic schools, from the NCEA and from the School 
Survey: Round 1 







School Survey 




1987-88 estimate of: 


NCEA 


Preliminary 


Final 


Number of Catholic schools 


8,992 


9,540 


9,527 


Enrollment (thousands) 


2,623 


2,827 


2,823 



SOURCE: Smith and Salvucci (1989). 



2.8.2 Prepublication review and evaluations of estimates, Round 2 

In Round 1, the discovery at the weighting stage of discrepancies between SASS preliminary 
estimates and CCD counts of schools and teachers led to substantial unanticipated processing 
costs, and significant delays in publication of the survey results. Changes were introduced in 
Round 2 in an attempt to eliminate or minimize the impact of such problems. The CCD replaced 
the QED list as the primary source of the frame for the public school sample. Instructions were 
added to the public school questionnaire to report data only for the school named on the label, 
and the expected numbers of teachers and students for each school were displayed on the label. 
Expected and reported school enrollment and teacher counts were compared in the field office 
edit operation, with followups for differences of 50 percent or more. 

In spite of these changes, initial postprocessing comparisons of weighted estimates by state 
showed that the SASS, estimates of total teachers from the public school data file were at least 15 
percent greater than the state FIE teacher counts from the 1991 CCD for 9 states, and staff 
reviews identified significant data problems for 1 additional state. For these 10 states, 
approximately 375 schools with large differences between the SASS and CCD records were 
identified. The individual records for these schools were compared, and when appropriate, the 
SASS records were changed to make them consistent with the CCD data for the schools. 
Changes were made to about 300 of the SASS records in this group. These comparisons of 
records for individual schools showed that there were two main causes of the SASS 
overestimates of teachers: some schools reported data for all of the schools in a school district, 
and there were instances where 2 or more schools (as defined for CCD) at a single location were 
reported as a single school. 
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After the changes were made, the school files for the 10 states were reprocessed to produce new 
estimates of students and teachers. Some residual problems were identified and a few additional 
changes were made. As will be noted in chapter 5, for the schools for which changes had been 
made in the postprocessing edit, some of the teacher records also required changes. (For 
additional details, see chapter VII, section F of the Data File User's Manual for Round 2.) 

Round 2 evaluation of published estimates. Final estimates of public schools by state for Round 
2 of the School Survey conducted in 1990-91 were compared with school counts from the CCD 
for the same school year. The SASS estimate for 1 state, Oklahoma, was about 15 percent higher 
than the CCD count. There were differences of 5 to 10 percent for 8 states and the District of 
Columbia. The SASS estimates were from 5 to 10 percent higher for Arizona and Nevada and 5 
to 10 percent lower for Alaska, Louisiana, Minnesota, South Dakota, Wisconsin, Wyoming, and 
the District of Columbia. For the remaining 42 states, the SASS estimates were within 5 percent 
of the CCD counts. For the United States, the SASS estimate was 97.9 percent of the CCD count 
for the same school year (Gruber, Rohr, and Fondelier, 1993, table XII-4). 

The CCD counts may be expected to be higher mainly for two reasons. With a few exceptions, 
the SASS sample did not cover schools that did not exist at the time of the 1989 CCD but were 
reported in the 1991 CCD. Second, 4 percent of the schools sampled from the 1989 CCD were 
found in SASS to be ineligible for the survey because they were no longer operating, had merged 
with another school, or were not serving students in any of grades 1-12. A factor causing 
differences in the other direction was that some of the schools sampled from the 1989 CCD list 
were found in SASS to represent more than one school, for example, an elementary and 
secondary school operating at the same location but under separate administration. Although the 
elimination of schools that were ineligible during the reference school year and the inclusion in 
the sample of units with more than one school in SASS give rise to differences between SASS 
and CCD estimates, these factors did not cause any bias in the SASS estimates. 

The SASS estimates of the numbers of private schools by school type (nine categories of schools 
with religious affiliations and nonsectarian schools) were compared with counts from the 
1989-90 PSS, which provided the sampling frame for the sample of private schools for the 
Round 2 School Survey (which covered the 1990-91 school year). The SASS estimates were 
smaller in all nine categories. This was primarily the result of 1989-90 PSS schools that were 
found to be out of scope in SASS in 1990-91. As mentioned before, 5.6 percent of the sample of 
private schools were found to be ineligible because they were no longer operating, had fewer 
than 10 students, or did not meet the SASS definition of a school for other reasons. For the 
United States, the SASS estimate of the number of private schools came to 92.4 percent of the 
1989-90 PSS count (Gruber, Rohr, and Fondelier, 1993, table XII-5). 

For private schools, the Round 2 School Survey estimates of numbers of students and teachers 
were compared with counts from the 1989-90 PSS. The SASS student counts were 3.8 percent 
higher than the PSS counts and the SASS teacher counts were 1.7 percent lower. These 
differences may have been due in part to the sampling error associated with the SASS estimates. 

The 1989-90 PSS enrollment counts for Catholic schools exceeded counts from NCEA's census 
for the same year by 5.4 percent of the latter's figure (Gruber, 1992a). Differences by state 
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showed large variations; however, some of these could be attributed to the inclusion by the 
NCEA in a single state of counts for archdioceses with schools, in more than one state. 

Data reported on the School Survey questionnaires were not always internally consistent. For 
example, the total of enrollment counts by grade frequently differed from the total of enrollment 
counts by race/ethnicity for the same school. The questionnaire item on enrollment by grade 
asked for counts as of October 1, whereas the item on enrollment by race/ethnicity did not 
specify a reference date. For schools with large discrepancies, the data for the two items were 
edited to make them consistent, but there were some residual differences. 

2.8.3 Evaluation of estimates, Round 3 

The prepublication review of Round 3 identified the need for less extensive editing than previous 
rounds, and comparisons of SASS estimates with estimates from other sources showed a fairly 
high level of agreement. These findings suggest that changes introduced into the Round 3 
questionnaires and procedures were successful. The SASS Data File User's Manual (Gruber et 
al., 1996) provides a detailed account of comparisons with other sources. 

Public school estimates. Gruber et al. (1996) compare Round 3 (1993-94) SASS estimates of the 
numbers of public schools with the numbers of public schools on the 1991-92 CCD Public 
School Universe file (excluding CCD schools that are out of scope for SASS, such as schools 
that have only prekindergarten and kindergarten levels only, schools with only basic adult 
education, or vocational schools with only postsecondary studies). Nationally, the SASS number 
for public schools was about 2 percent lower than the CCD number (see table 2-21). Only the 
District of Columbia and Nebraska had estimated numbers of public schools from SASS that 
were below 90 percent of the CCD numbers. Four states had SASS estimates of their numbers of 
schools between 90 and 95 percent of the CCD numbers, and there were four states in which 
SASS estimates were higher than the CCD numbers (all were within 1 percent of the CCD 
numbers). Hoffman (1995) discusses the changes in school estimates between the 1991-92 and 
the 1993-94 CCDs. 

The estimated numbers of public school students in the Round 3 (1993-94) School Survey were 
compared with the estimates for the same school year from the CCD State Nonfiscal Survey 
(Johnson, 1995). Two comparisons were made, one to the CCD total number of students, and 
the CCD count of kindergarten to grade 12 students. The latter is the better comparison for 
SASS, since the SASS definition of students (i.e., students in kindergarten to grade 12 for 
schools that offer a grade 1 as well as kindergarten) excludes prekindergarten students. 

Overall, the estimated SASS student count is almost 4 percent lower than CCD's total students 
and just over 3 percent lower than CCD's kindergarten to grade 12 student count (see table 2-22). 
There are slightly more than one-half a million prekindergarten students included in CCD, and 
excluding them brings the SASS student count into the same degree of "fit" as the SASS number 
of schools (a difference of about 3 percent). Excluding prekindergarten students of course 
enlarges the difference in those states for which the SASS estimate is higher than the full CCD 
count, but in most cases the SASS estimate is only about 1 to 2 percent points higher than the full 
count. 
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Table 2-21.— Public schools in SASS School Survey and in the 1991-92 Common Core of Data 
(CCD), by state: Rounds 



State 


Number of schools 
1991-92 CCD* 


1993-94SASS 


SASS/CCD(percent) 


50 states and DC 


82,772 


80,740 


98 


Alabama 


1,287 


479 


99 


Alaska 


490 


478 


98 


Arizona 


1,072 


1,057 


99 


Arkansas 


1,092 


1,084 


99 


California 


7,549 


7,319 


97 


Colorado 


1,350 


1,329 


' 98 


Connecticut 


975 


964 


99 


Delaware 


172 


169 


98 


District of Columbia 


178 


160 


90 


Florida 


2,460 


. 2,348 


95 


Georgia 


.1,721 


1,723 


100 


Hawaii 


236 


234 


99 


Idaho 


583 


573 


98 


Illinois 


4,095 


3,884 


95 


Indiana 


1,891 


1,869 


99 


Iowa 


1,552 


1,518 


98 


Kansas 


1,458 


1,450 


100 


Kentucky 


1,384 


1,327 


96 


Louisiana 


1,445 


1,446 


100 


Maine 


730 ' 


721 


99 


Maryland 


1,199 


1,185 


99 


Massachusetts 


1,731 


1,689 


98 


Michigan 


3,295 


3,159 


96 


Minnesota 


1,584 


1,492 


94 


Mississippi 


965 


957 


99 


Missouri 


2,081 


2,082 


100 


Montana 


900 


890 


99 


Nebraska 


1,453 


1,296 


89 


Nevada 


370 


365 


99 


New Hampshire 


445 


445 


100 


New Jersey 


2,258 


2,195 


97 


New Mexico 


672 


663 


99 


New York 


3,942 


3,904 


99 


North Carolina 


1,943 


1,927 


99 


North Dakota 


616 


582 


94 


Ohio 


3,746 . 


3,636 


97 


Oklahomia 


1,809 


1,763 


97 


Oregon 


1,190 


1,184 


100 


Pennsylvania 


3,235 


3,128 


97 


Rhode Island 


307 


295 


96 


South Carolina 


1,096 


1,081 


99 


South Dakota 


670 


661 


99 


Tennessee 


1,512 


1,522 


101 


Texas 


5,972 


5,890 


99 


Utah 


704 


674 


96 


Vermont 


336 


318 


95 


Virginia 


1,800 


1,698 


95 


Washington 


1,835 ' 


1,806 


98 


West Virginia 


970 


898 


93 


Wisconsin 


1,999 


2,014 


101 


Wyoming 


417 


411 


99 



♦CCD schools that are out of scope for SASS are excluded. 
SOURCE: Gruber et al. (1996). 
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Table 2-22. — Public school students in the School Survey and in 1993-94 Common Core of Data 
(CCD), by state: Round 3 



State 


Number of students (in thousands) 

CCD CCD 

1993-94 lessPK SASS 


SASS/CCD (percent) 


SASS/CCD less PK 
(percent) 


50 states and DC 


43,476 


42,919 


41,622 


96 


97 


Alabama 


734 


726 


746 


102 


103 


Alaska 


126 


123 


127 


101 


103 


Arizona 


709 


706 


686 


97 


97 


Arkansas 


444 


443 


460 


104 


104 


California 


5,329 


5,267 


4,805 


90 


91 


Colorado 


625 


618 


616 


99 


100 


Connecticut 


496 


490 


473 


95 


96 


Delaware 


106 


105 


108 


102 


103 


District of Columbia 


81 


75 


76 


94 


101 


Florida 


2,041 


2,006 


1,889 


.93 


94 


Georgia 


1,235 


1,230 


1,194 


97 


97 


Hawaii 


180 


180 


173 


96 


96 


Idaho 


237 


235 


.218 


92 


93 


Ulinois 


1,893 


1,851 


1,748 


92 


94 


Indiana 


966 


962 


973 


101 


101 


Iowa 


499 


493 


484 


97 


98 


Kansas 


. 458 


455 


432 


94 


95 


Kentucky 


655 


640 


693 


106 


108 


Louisiana 


801 


788 


791 


99 


100 


Maine 


217 


216 


208 


96 


96 


Maryland 


773 


755 


- 754 


98 


100 


Massachusetts 


878 


865 


776 


88 


90 


Michigan 


1,599 


1,588 


1,492 


93 


94 


Minnesota 


810 


804 


705 


87 


88 


Mississippi 


506 


504 


532 


105 


106 


Missouri 


876 


852 


939 


107 


no 


Montana 


163 


163 


176 


108 


108 


Nebraska 


285 


282 


248 


87 


88 


Nevada 


236 


235 


231 


98 


99 


New Hampshire 


185 


184 


175 


94 


95 


New Jersey 


1,151 


1,142 


1,098 


95 


96 


New Mexico 


322 


320 


323 


100 


101 


New York 


2,734 


2,702 


2,594 


95 


96 


North Carolina 


1,133 


1,125 


1,091 


96 


97 


North Dakota 


119 


119 


116 


97 


98 


Ohio 


1,807 


1,790 


1,816 


101 


101 


Oklahoma 


604 


599 


580 


96 


97 


Oregon 


517 


516 


479 


93 


93 


Pennsylvania 


1,744 


1,740 


1,805 


104 


104 


Rhode Island 


146 


145 


124 


85 


86 


South Carolina 


644 


636 


630 


98 


99 


South Dakota 


143 


142 


140 


98 


98 


Tennessee 


867 


857 


841 


97 


98 


Texas 


3,608 


3,488 


3,343 


93 


96 


Utah 


. 471 


469 


454 


96 


97 


Vermont 


103 


. 101 


92 


89 


91 


Virginia 


1,045 


1,042 


958 


92 


92 


Washington 


916 


911 


913 


100 


100 


West Virginia 


314 


310 


316 


101 


102 


Wisconsin 


844 


827 


881 


104 


107 


Wyoming 


101 


101 


102 


102 


102 



SOURCE: Gruber etal. (1996). 
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There are 7 states in which the SASS public school kindergarten to grade 12 student count is 
lower than CCD's by more than 8 percent: California, Massachusetts, Minnesota, Nebraska, 
Rhode Island, Vermont, and Virginia. Five states have a SASS student count that is 5 percent or 
higher than CCD’s kindergarten to grade 12 student count: Kentucky, Mississippi, Missouri, 
Montana, and Wisconsin (Gruber et al., 1996). 

Private school estimates. For private schools, the 1993-94 SASS School Survey estimates were 
compared with the 1991-92 PSS (the frame year) and the 1993-94 PSS (the survey reference 
year). Table 2-23 shows the results of the comparisons by private school types. For the total 
estimate, there is no difference between the three surveys, part of this is due to the weighting 
adjustment that matched the 1993-94 SASS total estimate of private schools to the 1993-94 PSS 
total. By type of school, there is some variation in the ratio of the SASS estimate to the PSS 
1991-92 estimates. For Catholic schools, the PSS estimates are in fact larger by about 6 percent; 
whereas for other religious schools and nonsectarian schools, the SASS estimates are larger by 4 
percent and 3 percent respectively. Relative to the 1993-94 PSS estimates, the SASS estimates 
are virtually the same for Catholic, other religious, and nonsectarian schools. For some school 
subgroups, such as Catholic private order schools, other religious affiliated schools, and 
nonsectarian regular program and special education schools, the SASS estimates are larger than 
the PSS estimates by 6 percent or more. On the other hand, for Catholic and nonsectarian special 
emphasis schools, the SASS estimates are smaller by 10 percent or more. 



Table 2-23. — Private schools in the School Survey and in the 1991-92 and 1993-94 Private 
School Survey, by type: Round 3 



Private school type 


PSS 

1991-92 

number 


PSS 

1993-94 

number 


SASS 

1993-94 

number 


SASS/ 
1991-92 PSS 
(percent) 


SASS/ 1993- 
94 PSS 
(percent) 


All private schools 


25,998 


26,093 


26,093 


100 


100 


Catholic 


8,889 


8,331 


8,358 


94 


100 


Parochial 


5,485 


5,127 


5,332 


97 


104 


Diocesan 


2,502 


2,371 


2,133 


85 


90 


Private order 


901 


833 


893 


99 


107 


Other religious 


11,760 


12,222 


12,232 


104 


100 


Conservative Christian 


4,291 


4,530 


4,524 


105 


100 


Affiliated 


3,950 


3,640 


3,881 


98 


107 


Unaffiliated 


3,519 


4,051 


3,827 


108 


94 


Nonsectarian 


5,349 


5,541 


5,503 


103 


99 


Regular program 


2,376 


2,198 


2,342 


99 


107 


Special emphasis 


1,810 


2,106 


1,809 


100 


86 


Special education 


1,163 


1,237 


1,353 


116 


109 



SOURCE: Gruber a al. (1996). 



Table 2-24 shows the comparison of estimated number of full-time equivalent teachers (FTEs) 
between the 1993-94 SASS and the 1993-94 PSS. The 1993-94 SASS estimate of the number of 
private FTE teachers is about 2.5 percent lower than the 1993-94 PSS number for all private 
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schools. By types of private schools, the SASS estimates fall short of the PSS estimates for 
Catholic schools, other religious affiliated schools, and in particular, for nonsectarian special 
emphasis schools; the SASS estimate is below by more than 12 percent for these types of private 
schools. On the other hand, the SASS estimates exceed the PSS estimates for some school types. 
For example, for the other religious unaffiliated schools, the SASS estimate exceeds the PSS 
estimate by more than 13 percent. 

Table 2-24. — Number of full-time equivalent (FTE) teachers in private schools from the School 
Survey and the 1993-94 Private School Survey: Round 3 



Private school type 


Number of FTE teachers 
PSS SASS 


SASS/PSS (percent) 


All private schools 


339,267 


330,839 


98 


Catholic 


143,214 


132,240 


92 


Parochial 


75,839 


68,105 


90 


Diocesan 


42,239 


41,174 


97 


Private order 


25,136 


22,961 


91 


Other religious 


117,397 


120,254 


102 


Conservative Christian 


42,178 


44,841 


106 


Affiliated 


46,511 


42,839 


92 


Unaffiliated 


28,708 


32,574 


113 


Nonsectarian 


78,655 


78,345 


100 


Regular 


48,539 


49,533 


102 


Special emphasis 


16,552 


14,548 


88 


Special education 


13,564 


14,264 


105 



SOURCE: Gruber etal. (1996). 
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3. THE SCHOOL PRINCIPAL SURVEY 

3.1 INTRODUCTION 

The School Principal Survey (called the School Administrator Survey in the 1987-88 and 1990-91 
Rounds) is the second in the series of surveys comprising the Schools and Staffing Surveys 
(SASS). It follows directly from the School Survey, since the school principals in the sampled 
schools are the School Principal Survey respondents. The School Principal Survey gathers 
information about principals, about school staffing and program decisions, and about 
administrative practices. The questionnaire collects demographic information for the principals 
and also information on their academic background and experience, professional training, career 
plans, and attitudes toward school management issues. Information collected in the School 
Principal Survey can be integrated with the information collected in the School Survey and the 
Teacher Survey to fill critical gaps in the knowledge base about schools, teachers, and students. 
School principalsi who consist of principals, headmasters or headmistresses, were asked to 
complete their own questionnaires, rather than delegating all or part of the task to other staff, as 
was permitted for the School Survey questionnaire. 

For the most part, this chapter does not repeat the design features and procedures already described 
in chapter 2. This chapter has sections covering the main phases of the survey: sampling frames 
(3.2); sample design (3.3); content (3.4); data collection procedures (3.5); data processing (3.6); 
measurement error (3.7); nonresponse (3.8); estimation (3.9); and evaluation of estimates (3.10). 

3.2 SAMPLING FRAMES 

For all three survey rounds (Round 1 in 1987-88, Round 2 in 1990-91, and Round 3 in 1993-94), 
the target population for the School Principal Survey consisted of the principals or head 
administrators of all public and private schools eligible for inclusion in the School Survey. A small 
number of the sampled schools, primarily those with low enrollment or in rural areas, did not have 
principals. Schools were identified as not having a principal if the response box for the statement 
"This school has no principal/school head" was checked on the first page of the School Principal 
Survey questionnaire for private schools or if the response box for the same statement but without 
the words "school head" was checked on the questionnaire for the public schools. Schools 
indicating the absence of a principal were dropped from the School Principal Survey. In both 
Rounds 1 and 3, about 1 percent of the public schools and under 5 percent of the private schools 
had no principal, according to published survey estimates (Choy, Medrich, Henke and Bobbitt, 
1992; Henke, Choy, Geis, and Broughman, 1996). Except for such cases, the sampling frame for 
the School Principal Survey was the same as the school sampling frame, which is described in 
detail in section 2.4. Thus, the frame problems at the school level noted in section 2.4.3, such as 
undercoverage, duplicate schools, definitional changes, survey reference periods differing from 
frame reference periods, incomplete sampling information, and out-of-scope listings apply equally 
to school principals. 

Out-of-date or incorrect contact information, which can result in nonresponse, reflects another type 
of frame deficiency. Fortunately, only 0.2 percent of principals in Round 1 and 0.4 percent of 
principals in Round 2 could not be contacted. In Round 3, only 0.1 percent of both public and 
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private sampled school principals could not be contacted (Monaco, Salvucci, Zhang, Hu, and 
Gruber, 1998). These rates are upper limits for the cases not contacted because of incorrect contact 
information since there are other possible reasons for noncontact, such as a principal being away 
from school for the entire collection period. 

3.3 SAMPLE DESIGN 

For all three survey rounds, once the sample of schools is selected, no additional sampling is 
needed to select the sample of school principals. The principals' selection probabilities are those of 
their schools. Thus, principals are selected, within strata, with probabilities proportionate to the 
square root of the numbers of teachers as reported on the sampling frame for their schools. A 
detailed discussion of sampling procedures for schools is provided in section 2.4. 

Salvucci, Weng, and Kaufman (1995) conducted a detailed analysis of the design effects associated 
with the variances of a set of estimates from the School Principal Survey for Round 2. For this 
survey these design effects reflect increases in the variances arising from departures from an equal 
probability sample and decreases from the use of stratification and the estimating procedures. 
Large design effects indicate that sampling variability is high compared to that from a simple 
random sample of the same sample size. 

Salvucci et al. (1995) report average design effects for a set of 34 estimates of national totals (e.g., 
the total number of principals with a master's degree, the total numbers reporting certain types of 
school problems as serious) of 1.77 for public schools and 1.95 for private schools. The average 
design effect for a set of 10 estimates of national proportions (e.g., the proportion of principals 
reporting excellent teaching staff) was 1.78 for public schools and 2.37 for private schools. The 
inverse of the design effect represents the multiplier to be applied to the actual sample size with the 
complex sample desigii to give the effective sample size, where the effective sample size is the size 
of a simple random sample that would yield an estimate of the same level of precision. Thus, a 
design effect of 1.77 implies that the effective sample size is only 56 percent of the actual sample 
size. The cause of this reduction from the actual to the effective sample size lies mainly with the 
unequal selection probabilities used for sampling school principals; they were sampled via their 
schools which were selected with probability proportional to the square root of the number of 
teachers in the school, and sampling fractions varied markedly by domain, particularly by state, for 
public schools and by affiliation group for private schools. The reductions in effective sample sizes 
for national estimates implied by the above design effects thus mainly reflect the multiple 
objectives for SASS and the inevitable compromises in sample design that need to be made. 

3.4 CONTENT 

The content areas of the School Principal Survey questionnaire remained generally the same across 
rounds, although there were some changes in questionnaire items. The questionnaires are designed 
to obtain information about: 

■ The training, experience, professional background, salary and benefits, career plans, and 
demographic characteristics of school principals; 

■ School and staffing program decisions; 
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■ Administrative practices; 

■ The types of school problems that principals view as serious; and 

■ Their influences on school matters. 

The similarity of the questionnaires across rounds facilitates comparisons across time, one of the 
interests in SASS. Nevertheless, care must be exercised when interpreting change estimates 
between rounds, since question wording, ordering, context, or concepts may have changed. 

A single questionnaire was used in Round 1 for all school principals, but separate questionnaires 
were used for principals of public and private schools in Rounds 2 and 3 (most of the data items on 
the two versions were the same). A third questionnaire, which had essentially the same content as 
the public school version of the questionnaire, was added in Round 3 for school principals in BIA 
schools. 

Table 3-1 indicates the subject areas covered in the School Principal Survey questionnaire and the 
emphasis given to each area, by round. It also provides a measure of the changes in emphasis over 
time; for example, the number of items focussed on the educational background of the principal 
declined between Rounds 1 and 2 but increased dramatically in Round 3. New in Round 2 were 
questions dealing with perceptions of school educational goals and the quality of the teaching staff; 
also several new items were added to the list of school problems whose relative seriousness the 
principals were asked to evaluate. In Round 3 there were substantial increases in emphasis on the 
principal's administrative and other job experience and items dealing with his or her perceived 
influence on school matters. 

Some questions were modified and others were deleted between rounds in response to reinterview 
results from the previous rounds, to pretest results, or to previous survey results. For Round 2, 
some of the changes made to the School Principal Survey questionnaires were based at least in part 
on findings from a review of 600 questionnaires from a pretest conducted in school year 1989-90 
(Jefferson-Copeland and Bynum, 1990): the format and placement of codes for major and minor 
fields of study at the bachelor's and master's levels were revised, and skip patterns for a question on 
retirement plans were introduced to make clear that the second part of the question was not 
applicable to all respondents. Some questions were deleted between rounds because they were 
found to be unreliable in the reinterview programs or because they were found not to be useful. 

Questions in Round 1 that were dropped in Round 2 included a breakdown of time spent by the 
principal on different kinds of school-related activities, programs for teacher evaluation and 
assistance to beginning teachers, and problems encountered in filling vacancies. A multiple 
response "mark all that apply" question asking about methods of compensating for unfilled teacher 
vacancies also was dropped from Round 2, due to the low response rate achieved in Round 1. The 
Round 2 question asking the principal to rate the school's teaching staff was not included in Round 
3 because the principal's rating was an overall average for the school and this was felt by the 
Technical Review Panel to be too broad a measure of teaching quality. 

For Round 2 it was estimated (as printed on the questionnaire) that it would take the principals an 
average of about 20 minutes to complete the questionnaires, and this increased to 30 minutes for 



Table 3-1. — Overview of the questionnaire items in the School Principal Survey: Rounds 1-3 



Categories of questions 


Specific types of questions asked 


Number of items 

Round 1 Round 2 

(1987-88) (1990-91) 


1 

Round 3 
(1993-94) 


Total 




108 


105 


173 


Education and training 


Degrees achieved and major fields of study 


25 


18 


^9 




Professional training 


4 


7 


6 


Professional experience 


Teaching experience: years and assignment 

fip.lHc 


6 


4 


4 




Administrative experience: years and positions 


5 


5 


^24 




Other job experience 


6 


6 


^7 


Career plan 


Plan to remain as principal 


0 


2 


6 


Compensation 


Salary 


2 


2 


2 




Benefits 


10 


10 


10 


Demographics 


Gender, age. race/ethnicity 


4 


5 


5 


Job-related activities 


Activities and hours spent 


11 


0 


0 


Perceptions 


Perceptions of school problems 


13 


22 


24 




Perceptions of influence on school matters 


9 


15 


*39 




Perceptions on school educational goals 


0 


3 


3 


Questions about schools 


Teacher evaluation 


1 


3 


0 


Teaching staff 


Teacher training 


1 


0 


0 




Teacher recruitment 


8 


0 


0 


Miscellaneous 


Respondent name and telephone number 


3 


3 


4 



^Refers to total number of response items. A question may have multiple response items. 

^BA/BS degree granting university and location were added. 

^Grade level of previous principal positions and breaks in principal career were added. 

^New position categories and years of experience were added. 

^e private school version had 27 items. 

SOURCE: Zheng (1996). 

Round 3. A question was added to the questionnaire in Round 2, and retained in Round 3, to 
ascertain how long it actually took principals to complete the questionnaire. 

As part of a Round 3 study on ways to improve the mail return rates in SASS surveys. Cole, 
Palmer, and Schwanz (1997) examined the effect of response burden on mail response rates. They 
found that mail response rates decreased for each additional form respondents were asked to 
complete, suggesting a response burden effect. They recommended some types of nonmonetary 
incentives for respondents who complete several questionnaires. Although this was a statement 
about SASS in general, it may be particularly applicable to school principals who have to respond 
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to their own questionnaire, as well as possibly the school questionnaire, in addition to sending 
teacher lists to the U.S. Census Bureau. In Round 3, teaching principals were also eligible for 
selection into the teacher sample, thus increasing their potential for higher response burden. 
Additionally, the Student Records Survey questionnaires and the Library Survey questionnaires 
were addressed to the principal, although other staff members could be asked to complete these 
questionnaires. 

3.5 DATA COLLECTION PROCEDURES 

The principal or school head of each school selected for the School Survey was the only eligible 
respondent for the School Principal Survey. Collection procedures, which were essentially the 
same over the three rounds, are summarized in table 3-2. Generally, there was an advance mailing 
in late summer/early fall to LEAs and school principals to explain the nature of the SASS data 
collection activities and, in the case of principals, to ask them to submit a teacher list for use in 
selecting the sample of teachers for the Teacher Survey. This was followed by a mailing of the 
questionnaires for both the School Survey and the School Principal Survey in late fall or early 
winter. A second mailing of School Principal Survey questionnaires was sent to those who had not 
yet responded 5 or 6 weeks after the initial mailing. Telephone followup started 4 weeks after the 
second mailing for Round 1 and about 6 weeks after the second mailing for Rounds 2 and 3. 

Table 3-2. — ^Data collection procedures for the School Principal Survey: Rounds 1-3 



Procedure 


Round 1 
(1987-88) 


Round 2 
(1990-91) 


Round 3 
(1993-94) 


Advance mailing 


Fall, 1987 


Sept. 1990 


Aug. 1993 


Initial mailing 


Late Jan.-Feb„ 1988 


Dec. 1990 


Oct. 1993 


Reminder postcards 


None 


None 


1 week after initial 
mailing 


Second mailing 


6 weeks after initial 
mailing 


5 weeks after initial 
mailing 


5 weeks after initial 
mailing 


Followup 


By telephone, 4 weeks 
after second mailing 


By telephone from 12 
regional Census offices, 
6*7 weeks after second 
mailing 


By CATI from 2 
Census centralized 
locations, 6 weeks after 
second mailing 


Quality assurance/control 


— 


Regional office reviews of 
completed questionnaires 


CATI edits and 
automated skip patterns 



~Not available. (Data were not collected or not reported.) 

SOURCES: NCES (1991b); Gruber, Rohr, and Fondelier (1993, 1996). 



Two innovations were introduced in Round 3 following field testing in 1991-92. Reminder 
postcards were mailed to principals 1 week after the initial mailing of questionnaires, and a toll-free 
U.S. Census Bureau telephone number was established to enable principals to call for assistance in 
completing the questionnaire. The extent of use of the toll-free number is not known. 
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Overall, mail return rates for principals for Round 3 (75 percent for public school, 67 percent for 
private school, and 67 percent for BIA school principals) were about 10 percent higher than for 
Round 2 (67 percent for public school and 55 percent for private school principals). This increase 
in mail response rates in Round 3 may have been due to the introduction of the postcard, and the 2 
weeks added to the response period before interview followup began. 

In Rounds 1 and 2, followup of mail nonrespondents was conducted by field representatives in U.S. 
Census Bureau regional offices, who attempted to collect the data by telephone using a paper-and- 
pencil questionnaire. The Round 3 followup used a computer-assisted telephone interviewing 
(CATI) approach from two U.S. Census Bureau centralized locations. Table 3-3 shows that the 
CATI followup resulted in the completion of about 86 percent of the followup cases for both public 
and BIA school principals, and of about 68 percent for private school principals (see table 3-3). 
Field representatives in the U.S. Census Bureau regional offices dealt with cases by telephone for 
which CATI interviewers were unable to collect the data due to incorrect phone numbers (2 percent 
of public school followup cases, 4 percent of private school followup cases, and 1 percent of BIA 
school followup cases). Among the cases assigned to them, the field representatives successfully 
completed questionnaires for more than a third of the public and private school principals and all of 
the BIA school principals. 

Table 3-3. — Principals requiring and completing telephone interview followup, by type of school 
and followup interview: Round 3 



CATI followup Non-CATI followup 

Total Completed Total Completed 



School type 


Number 


Percent* 


Number 


Percent^ 


Number 


Percent^ 


Number 


Percenr 


Public 


2,410 


25 


2,072 


86 


158 


2 


55 


35 


Private 


1,108 


33 


749 


68 


125 


4 


55 


44 


BIA 


53 


33 


47 


89 


1 


1 


1 


100 



^Percent of sample cases in CATI followup. 

^Percent of sample cases in CATI followup that were completed. Cases identified as noninterviews and out of scope for the survey during the CATI 
followup are not included. 

^Percent of sample cases in telephone followup by field representatives. 

^Percent of sample cases in telephone followup by field representatives that were completed. Cases identified as noninterviews and out of scope for 
the survey during followup are not incluc^. 

NOTE: Numbers do not total because of noninterviews and out-of-scope cases and because nonresponse cases may be switched between CATI and 
non-CATI followup. 

SOURCE: Gruber etal. (1996). 



The sizable amount of telephone followup in all rounds (around 30 percent) has raised concern 
about possible lack of comparability of responses due to different collection modes (Monaco et al., 
1998). However, to date, no studies have been initiated to examine this issue. 

Quality assurance. As in the case of the School Survey, in Rounds 1 and 2 the primary method of 
controlling the quality of the School Principal Survey data was through regional office reviews of 
questionnaires completed by U.S. Census Bureau field representatives in their telephone followups 
of mail nonrespondents. The procedures for the regional office reviews were identical to those 
used for the School Survey, as described in section 2.6.1. During the data collection for Round 2, 
regional office staff reviewed a sample of followup questionnaires completed by telephone 
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followup (Pasqualucci, 1991). An analysis of the forms used to record the results of these reviews 
showed that the main source of errors identified was a failure by several of the U.S. Census Bureau 
field representatives to record codes for major and minor degree fields. Also, a check item, 
designed to skip the next item on the questionnaire when it did not apply, had been left blank on 
several questionnaires. In Round 3, quality assurance also included the use of CATI capabilities, 
such as on-line data capture, coding, and some editing, as well as controlled skip patterns. 

3.6 DATA PROCESSING 

The sequence and nature of the data processing operations for the School Principal Survey in all 
rounds were essentially the same as those described for the School Survey in section 2.6. These 
included assigning a status to each questionnaire, performing a general clerical edit, assigning 
Federal Information Processing Standards (FIPS) codes for . the School Principal Survey 
questionnaires, grouping questionnaires for data entry, performing data entry with 100 percent 
independent verification, assigning the preliminary interview status, performing computer editing 
that checked ranges and within- and between-item data consistency, making corrections where 
correct responses could be determined, and assigning a final interview status code. For those 
questionnaires completed by CATI, several of the above operations, such as coding and capture, 
were done during the interview. The resulting CATI data files were merged with those that had 
been data captured from paper questionnaires and underwent the same processes as the paper 
questionnaires from that point onward. 

In Round 2, a computer pre-edit was also implemented. As noted in section 2.7.2, the computer 
pre-edit was designed to identify inconsistencies and invalid entries for key data items. This 
operation produced a list of problematic cases. These cases were reviewed, and respondents were 
recontacted if necessary. Any responses that were deemed to be erroneous but that could not be 
corrected were deleted. In this round, an on-line pre-edit correction system was tested for the 
School Principal Survey questionnaires, which was in line with plans to further automate SASS 
processing. A review of pre-edit reject rates, edit change tallies, and post-edit item response rates 
provided evidence of problems with skip patterns (Jenkins, 1992b). About 10 percent of the 
respondents or followup interviewers failed to provide a response to a check item leading to a skip 
pattern; the edit change tallies showed that respondents had apparently failed to follow all of the 
skip instructions and consequently answered some items that did not apply to them. There was no 
prC'Cdit for the School Principal Survey questionnaire in Round 3 because there were no major 
issues in inter-item inconsistencies to reconcile. The former pre-edit conditions were incorporated 
into the blanking edit of the computer edit procedure, and resulted in those error conditions being 
handled by imputation rather than manual review and correction. 

The counts of changes made to the School Principal Surveys' questionnaire items as a result of the 
computer edit provide one indication of items that may have been problematic for respondents. 
Changes consist of deleting entries outside of acceptable ranges, correcting inconsistencies, and 
deleting responses that could not be corrected. A review of the extent of changes by question 
suggests that in Round 3, questions about school positions held before becoming a principal, about 
the number of breaks in service, and about plans for retirement were problematic for all school 
principals. Public school principals appeared to have difficulty with an item that required them to 
look up and enter a code representing their main teaching assignment for the most recent teaching 
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year, while private school principals had considerable difficulty in indicating the year in which they 
would be eligible to retire. 

3.7 MEASUREMENT ERROR 

As discussed in section 2.6.3, reinterview studies have been a part of the data collection activities 
for SASS surveys for all three rounds. Reinterviews were conducted in all three rounds with a 
sample of about 10 percent of the school principals responding to the School Principal Survey and 
at least 80 percent of these were successfully completed in each round. 

In Round 1, a single reinterview questionnaire containing selected items from both the School 
Survey and School Principal Survey questionnaires was used. In the later two rounds, the 
reinterviews of school principals were conducted separately from the School Survey questionnaire 
reinterviews, using a subset of items only from the School Principal Survey questionnaire. 

In the first two rounds, all reinterviews were conducted by telephone about 2 to 3 weeks after the 
original interview without regard to the mode of collection for the original interview (Gruber et al., 
1996). In Round 3, the reinterview mode of collection was matched to the original mode of 
collection in order to reproduce the conditions of the original survey. The mail reinterviews 
occurred about 3 to 4 weeks after the original interview and the CATI reinterviews took place 
about 1 to 2 weeks after the original interview. 

Reinterview results are analyzed using the index of inconsistency diVid the L-fold index of 
inconsistency, as described in section 2.6.3. The former measures the percent of total variance for 
an item with a dichotomous response that is accounted for by response variance. The latter is used 
for closed response items with more than two response categories; it is a weighted average of the 
simple index over all categories. As a rough rule of thumb, response variance is considered to be 
low when the index of inconsistency is less than 20 percent, moderate when it is between 20 and 50 
percent, and high when it is greater than 50 percent. Table 3-4 shows reinterview results for all 
rounds. 



Table 3-4. — Number of School Principal Survey reinterview items, by round, type and index of 
inconsistency: Rounds 1-3 



Index of inconsistency 


Round r 
(1987-88) 




Round 2 
(1990-91) 




Round 3 
(1993-94) 




Factual Opinion 


Factual Opinion 


Factual Opinion 


Total questions 


11 


22 


26 


0 


43 


32 


Low (0-19%) 


1 


0 


5 


0 


29 


0 


Moderate (20-50%) 


4 


3 


10 


0 


11 


0 


High (51-100%) 


4 


19 


10 


0 


3 


32 


Not available* 


2 


0 


1 


0 


0 


0 



*Did not meet the minimum requirements to compute a reliable estimate of the index of inconsistency. 
SOURCES: Newbrough (1989); Royce (1992); Bushery, Schreiner, and Sebron (1998). 
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Almost half of the reinterview questions displayed high response variance in all rounds. Many of 
these were opinion or perception-type questions for which any discrepancies between the two 
interviews may be due in part to a real change in attitude or perception (Bushery et al., 1998). 
Nearly all the factual items in Round 3 had low or moderate response variance, and they appear to 
have performed better in this regard than the factual items in earlier rounds. However, it should be 
noted that results are not strictly comparable across rounds because different sets of questions and 
different methods were used for the reinterviews. 

As just noted, opinion items generally had high levels of response variance. In Round 1, none of 
the 22 opinion items evaluated had low indexes of inconsistency and most were in the high range. 
These 22 items were of two kinds: 

■ A set of 13 items that asked principals for their views of the relative importance in their 
schools, on a 4-point scale, of each of 13 different kinds of problems that occur in some 
schools. Three of these problem types — student pregnancy, student use of alcohol, and student 
drug abuse — had estimated indexes in the moderate (20 to 50 percent) range; the rest were in 
the high range. 

■ A set of 9 items that asked principals for their evaluation, on a 6-point scale, of the relative 
influence of teachers, principals, and governing bodies on policies for establishing curricula, 
hiring new teachers, and discipline. All of these had indexes in the high range. 

These two sets of opinion items were revised into an expanded form for the Round 2 School 
Principal Survey. However, they were not included in the Round 2 reinterviews, since it was 
decided that more information of value for question improvement through cognitive research and 
better questionnaire design would come from the reinterview results for factual items than from 
those for opinion items (Bushery, Royce, and Kasprzyk, 1992). In general, NCES decided to focus 
the Round 2 reinterview efforts on factual questions. The Round 3 reinterviews contained a single 
set of 32 opinion questions, all of which displayed high response variance. 

A topic of concern with regard to response variance relates to the principals' college degrees and 
major fields of study. This was the only topic included in the reinterviews for both Rounds 1 and 
2. As a consequence of the high response variance for this question in Round 1, the question 
format was substantially revised from the multiple response, "mark all that apply" format, so that in 
Round 2, separate sets of questions with Yes/No responses were asked about bachelor's and 
master s degrees. As a result, the index of inconsistency for master's degrees was lowered from 
49.4 in Round 1 to 11.3 in Round 2. The index for bachelor's degrees had a value of 98.5 in Round 
1, but is unavailable in Round 2. 

Another set of questions that raised concern about response variance relates to the principal's 
previous positions. In Round 2, the question, "What other school positions, if any, did you hold 
before you became a principal?" provided a list of six positions for principals to choose from. The 
index of inconsistency ranged from about 32 percent to 97 percent (Royce, 1994). In Round 3, the 
list of positions was presented in a Yes/No format for each position. As a result, the index of 
inconsistency was reduced for all six of the positions to within a range from about 12 to 33 percent 
(Bushery et al., 1998) and the reduction was statistically significant for four of them (Feindt, 
Schreiner, and Bushery, 1997). 
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3.8 NONRESPONSE 



Unit nonresponse. Weighted unit response rates by state for public school principals, and by 
association group for private school principals, for all rounds can be found in tables 3-5 and 3-6, 
respectively. Table 3-7 shows the weighted response rates for private school principals by type of 
private school. Sampled cases that were out of scope because the principal's school was not 
operating in the school year of reference for the survey or because the sampled school had no 
principal were excluded from the base of the response rates. In Round 2, these exclusions 
accounted for some 5 percent of sampled public school cases and 7 percent of sampled private 
school cases. In Round 3, they accounted for 4 percent, 9 percent, and 6 percent of sampled cases 
in the public, private, and BIA sectors, respectively. 

In Rounds 1 and 2, a few LEAs requested NCES not to ask sample schools in their districts to 
participate in SASS. The nonresponse to the School Principal Survey due to district refusals was 
less than 0.5 percent of the eligible school principals in both Rounds 1 and 2 (Nash, 1988), and 
nonexistent in Round 3. 

Tables 3-5 and 3-6 show that response rates have generally been high. Weighted response rates for 
principals in the public sector have not changed much over the three rounds, varying only between 
94 and 97 percent, whereas rates for the private sector increased by about 10 percent between the 
first and second rounds (from 79 to 90 percent), and remained at almost that level (88 percent) in 
the third round. In all rounds, the response rate for public school principals was higher than that for 
private school principals. In Round 3, the response rate for BIA school principals was about 99 
percent. 

The response rates for principals of public schools were similar to those for public schools in the 
School Survey although slightly higher (the latter varied between 92 and 95 percent for the three 
rounds). The variation in response rates for private schools in the school survey (79 to 84 percent) 
between the three rounds was not as large as that for their principals. The response rate for BIA 
principals was the same as the BIA school response rate in Round 3. 

There was substantial variation in response rates within each sector. For public school principals, 
31 states in Round 1, 44 states in Round 2, and 44 states in Round 3 had weighted response rates of 
at least 95 percent. In the first and third rounds, the District of Columbia had the lowest weighted 
public sector response rate, at about 69 percent in Round 1 and 86 percent in Round 3. In Round 2, 
Maryland had the lowest weighted response rate (82 percent). 

In the private sector, the range of weighed response rates by association group in Round 1 was 
from 56 to 98 percent. Most groups were in the range from 70 to 90 percent. In second and third 
rounds, with an expanded set of association groups, weighted response rates ranged from 72 
percent to 100 percent; 12 of the 19 groups in Round 3 had rates of at least 90 percent. In all 
rounds the response rates for private school principals sampled from the area frame were 
substantially lower than for those sampled from the list frame. This finding has also been observed 
in the Private School Survey (Gruber et al., 1996). 





Table 3-5. — Weighted response rates for the survey of public school principals, by state: 
Rounds 1-3 



Weighted response rate (percent)* Weighted response rate (percent)* 



State 


R.ound 1 
(1987*88) 


Round 2 
(1990*91) 


Round 3 
' (1993*94) 


State 


Round 1 
(1987-88) 


Round 2 
(1990*91) 


Round 3 
(1993*94) 


Total 


94 


97 


97 










Alabama 


98 


99 


99 


Montana 


98 


99 


96 


Alaska 


99 


97 


96 


Nebraska 


96 


98 


96 


Arizona 


99 


97 


95 


Nevada 


97 


98 


94 


Arkansas 


97 


97 


98 


New Hampshire 


99 


99 


100 


California 


92 


96 


95 


New Jersey 


95 


92 


96 


Colorado 


99 


98 


89 


New Mexico 


97 


99 


96 


Connecticut 


92 


97 


96 


New York 


89 


90 


93 


Delaware 


90 


94 


99 


North Carolina 


94 


.96 


98 


District of Columbia 


69 


89 


86 


North Dakota 


95 


99 


99 


Florida 


99 


94 


98 


Ohio 


97 


97 


96 


Georgia 


95 


95 


99 


Oklahoma 


90 


99 


95 


Hawaii 


85 


99 


96 


Oregon 


98 


97 


97 


Idaho 


97 


100 


99 


Pennsylvania 


92 


97 


96 


Illinois 


97 


99 


98 


Rhode Island 


99 


97 


94 


Indiana 


98 


100 


98 


South C^olina 


91 


99 


97 


Iowa 


96 


99 


99 


South Dakota 


100 


99 


99 


Kansas 


94 


98 


94 


Tennessee 


95 


98 


97 


Kentucky 


92 


99 


95 


Texas 


92 


98 


97 


Louisiana 


92 


94 


98 


Utah 


100 


99 


99 


Maine 


99 


98 


93 


Vermont 


98 


99 


94 


Maryland 


81 


82 


95 


Virginia 


94 


95 


96 


Massachusetts 


93 


97 


99 


Washington 


99 


94 


99 


Michigan 


99 


99 


98 


West Virginia 


96 


99 


100 


Minnesota 


95 


99 


99 


Wisconsin 


94 


97 


99 


Mississippi 


98 


98 


98 


Wyoming 


89 


96 


98 


Missouri 


90 


99 


98 











*The basic weight is used to reflect the probability of selection. 
SOURCES: NCES (1991b); Gruber, Rohr, and Fondelier (1993, 1996). 
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Table 3-6. — ^Weighted response rates for the survey of private school principals, by association 
group: Rounds 1-3 " 









Weighted response rate (percent)* 








Round 1 


Round 2 


Round 3 


Association group 






(1987-88) 


(1990-91) 


(1993-94) 


Total, area frame and list frame 






79 


90 . 


88 


Area sample 






66 


83 


75 


List sample 






— 


92 


88 


List sample by association group: 












Association of Military Colleges and Schools of U.S. 






92 


96 


89 


National Catholic Education Association, 






91 


96 


93 


Jesuit Secondary Education Association 






Friends Council on Education 






85 


94 


99 


National Association of Episcopal Schools 






88 


94 


86 


National Association of Hebrew Day Schools 


1 


1 




86 


86 


Solomon Schechter Day Schools 


1 


. 


72 


98 


98 


Other Jewish 








72 


78 


Lutheran Church - Missouri Synod 








97 


100 


Evangelical Lutheran Church - Wisconsin 




L 


88 


98 


95 


Evangelical Lutheran Church in America 




r 


99 


98 


Other Lutheran 


> 






97 


98 


General Conference of Seventh Day Adventists 






89 


95 


93 


Christian Schools International 






98 


94 


74 


American Association of Christian Schools 






V 56 . . 


73 


82 


National Association of Private Schools for Exceptional Children 




85 


95 


97 


American Montessori Society Schools, other Montessori 






79 


92 


90 


National Association of Independent Schools 






76 


94 


.90 


National Independent Private School Association 






i 


i 


100 


Other private schools 






73 


85 


82 



*Not applicable. (Not listed as a separate association group.) 

*The basic weight is used to reflect the probability of selection. 

“Not available. (Data not collected or reported.) 

SOURCES: NCES (1991b); Gruber, Rohr, and Fondelier (1993, 1996). 
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Table 3-7. — ^Weighted response rates for survey of private school principals, by type of private 
school: Rounds 2 and 3 





Weighted response rate (percent)* 




Round 2 


Round 3 


Private school type 


(1990-91) 


(1993-94) 


All private principals 


90 


88 


Catholic 


97 


92 


Parochial 


96 


92 


Diocesan 


98 


93 


Private order 


98 


89 


Other Religions 


85 


83 


Conservative Christian 


82 


83 


Affiliated 


91 


82 


Unaffiliated 


80 


84 


Nonsectarian 


90 


90 


Regular 


86 


91 


Special emphasis 


93 


89 


Special education 


95 


89 



♦The basic weight is used to reflect the probability of selection. 

SOURCES: McLaughlin. O’Donnell. Ries. Broughman (1995); McLaughlin and Broughman (1997). 



Analytic studies were conducted for the second and third rounds to examine variations in response 
rates for different categories of school. Some results are displayed in table 3-8. The study by 
Scheuren et al. (1996), which investigated variations in response rates for the first four 
characteristics in the table for Round 2, used a multiple comparison test with a 10 percent 
significance level for each characteristic. In the table, S denotes a significant difference in response 
rates between at least one pair of categories for the characteristic; NS denotes no significant 
differences. Some significant category differences were found by region and school size for both 
public and private schools, by urbanicity for public schools, and by school level for private schools. 
However, as the table shows, the response rate differences are not great, particularly in the case of 
public schools. 

Monaco et al. (1998) performed similar analyses for Round 3, including the four characteristics 
examined in the Round 2 study and adding the characteristics displayed in the lower part of table 3- 
8 (some of which relate only to public schools, others only to private schools). They used an 
overall test of differences across categories (Rao-Scott chi-squared statistic); the significance level 
reached is indicated by the P-value in the table. Differences were deemed significant if the P-value 
was smaller than 0.05. Round 3 exhibits a similar pattern of significance/nonsignificance for the 
first four characteristics as Round 2. 





85 



Table 3-8. — Weighted response rates for the Survey of School Principals, by selected 
characteristics: Rounds 1 and 3 



Weighted response rate (percent)* 



Round 2 Round 3 

(1990-91) (1993-94) 



Characteristics 


Public 


Private 


Public 


Private 


Region: 


West 


97 


91 


^ 95 


87 


South 


96 


86 


97 


84 


Northeast 


94 


91 


96 


86 


Midwest 


99 


92 


98 


93 




S^ 


S^ 


P=0.034 


P<0.001 


Urbanicity: 


Central city 


94 


90 


95 


89 


Urban fnnge/large town 


96 


94 


96 


87 


Rural 


99 


86 


98 


85 




S^ 


ns’ 


P=0.002 


P=0.156 


School level: 


Combined 


96 


84 


97 


78 


Secondary 


98 


94 


97 


93 


Elementary 


96 


93 


96 


92 




ns’ 


S^ 


P=0.074 


P<0.001 


School size: 


1-149 


97 


86 


98 


84 


150-499 


97 


94 


97 


91 


500-749 


97 


92 


97 


92 


750+ 


95 


93 


95 


92 


School type: 


S^ 


S^ 


P=0.005 


P<0.001 


Nonregular 


♦ 


♦ 


98 


t 


Regular 


♦ 


♦ 


97 

P=0.009 




Minority enrollment: 


<5.5% 


♦ 


♦ 


98 




5.5-20.5% 


♦ 


♦ 


97 


t 


20.5-50.5% 


♦ 


♦ 


97 


t 


>50.5% 


♦ 


♦ 


94 

P<0.001 


t 


School sampled with 


certainty: 


♦ 


♦ 


97 




Yes 

No 


♦ 


♦ 


97 

P=0.961 




School sampled in 1990-91: 


Yes 


♦ 


t 


97 


92 


No 


♦ 


t 


97 


87 


Submitted a teacher list: 






P=0.768 


P=0.004 


Yes 


♦ 


t 


97 


♦ 


No 

1991-92 PSS status: 


♦ 


t 


77 

PcO.OOl 


♦ 


Respondent 




t 


t 


89 


Nonrespondent 




t 


t 


48 


Not in 1991-92 PSS 


t 




t 


76 

P<0.001 



^Not applicable. 

‘The basic weight is used to reflect the probability of selection. 

indicates a significant difference between the response rates for at least some of the levels of the variable at the 10 percent significance level. 
^NS indicates there are no significant differences at the 10 percent significance level. 

SOURCES: Scheuren et al. (1996); Monaco et al. (1998). 
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The new characteristics added for the Round 3 analyses show a variation in response rate by 
minority status and school type for public schools, and also by response to other data collections in 
both the public and private sectors. The small proportions of principals in public schools who 
failed to provide teacher lists for SASS and in private schools who were nonrespondents to the 
Private School Survey had appreciably lower response rates in the School Principal Survey than 
principals in schools responding to those data collections. Principals in private schools sampled for 
Round 2 of SASS had a higher response rate (92 percent) than those in schools not sampled (87 
percent) in Round 2. This finding is not surprising, given overlap procedures between Rounds 2 
and 3, specified in the sample design: the sample overlap in Round 3 of private schools in 
affiliation groups with high response rates in Round 2 was 30 percent, whereas that in affiliation 
groups with low response rates was minimized (see section 2.3). 

Both studies extended their analyses to develop logistic regression models predicting response rates 
using the characteristics studied as explanatory variables. Details are presented in the reports. 
Since the differences in response rates by category are not that large, the models are not highly 
predictive. The results obtained from the two analyses are inconsistent; possible reasons are 
suggested by Monaco et al. (1998) (e.g., difference between the 1990-91 SASS sample and the 
1993-94 sample, different procedures used to fit the logistic regression models). 

Monaco et al. (1998) also examined patterns of response/nonresponse across the various SASS 
surveys in Round 3. One of their findings was that the response rate in the School Principal Survey 
for principals in public schools that responded to the School Survey (98 percent) was significantly 
higher than that for principals in public schools that did not respond to the School Survey 
(78 percent). 

Item nonresponse. Table 3-9 summarizes item response rates for all three survey rounds of the 
School Principal Survey. In making comparisons across rounds it needs, of course, to be 
recognized that the questionnaires changed between rounds (see section 3.4). Item response rates 
were highest in Round 2, with all items on the public school principal questionnaire and all but a 
few items on the private school principal questionnaire having item response rates of 90 percent or 
higher. 

The lowest item response rates in Round 1 (70 percent for public schools and 72 percent for private 
schools) were for a multiple response "mark all that apply" item concerning methods of 
compensating for unfilled teacher vacancies. The items with the lowest response rates in Round 2 
were items immediately following skip instructions, suggesting that some respondents may have 
misunderstood these instructions (Jenkins, 1992b). On the public and private school questionnaires 
in Round 3, a question that asked for school positions held before becoming a principal and for the 
number of years in those positions had several category response rates lower than 75 percent. 
Additional questions on the private school principal questionnaire that had response rates lower 
than 75 percent were: a question asking for the year in which the principal would become eligible 
to retire from his or her position as principal (74 percent response); a question asking for the year in 
which the principal plans to retire from his or her position as principal (64 percent response); and a 
question asking whether the principal, if American Indian or Alaska Native, was enrolled in a state 
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Table 3-9. — Percent of items with selected response rates for the Survey of School Principals, by 
sector: Rounds 1-3 



Sector and round 


Percent of items with response rates: 
> 90 percent <75 percent 


Minimum of item 
response rate 
(percent) 


Round 1 (1987-88) 


Public 


86 


2 


70 


Private 


89 


2 


72 


Round 2 (1990-91) 


Public 


100 


0 


90 


Private 


98 


0 


80 


Round 3 (1993-94) 


Public 


92 


4 


65 


Private 


90 


6 


55 


BIA 


91 


1 


72 



SOURCES; NCES (1991b); Gruber et al. (1996). 



or federally recognized tribe (54 percent response). On the BIA school principal questionnaire, 
only the "other" category part of the question asking for school positions held before becoming a 
principal and for the number of years in those positions had a response rate lower than 75 percent. 

3.9 ESTIMATION 

Imputation. Item nonresponse was handled in Rounds 2 and 3 by imputation. Imputation was not 
employed in Round 1 because of time and budget constraints in the processing cycle (Kaufman, 
1991). However, imputations were later done and can be found on the CD-ROM (NCES, 1998) 
containing data from all three rounds. 

In Round 1, some items were assigned values as part of the computer edit and in preceding 
operations based on other information available on the School Principal Survey or associated 
School Survey questionnaires. Missing responses after this stage were left blank. These missing 
responses were treated as zero values in the ED tabs and analytic reports, thus leading to an 
underestimation of totals. 

In Round 2, missing responses on the School Principal Survey questionnaire were imputed after 
completion of the imputation of missing responses on the School Survey questionnaire. The 
imputations were performed in a way that produced consistency between data from the two 
questionnaires. Certain items were common to both questionnaires, and the first step in the 
imputation process was to carry over values for these common items, whether reported or imputed, 
from the school records. Following this first step, imputation for the remaining missing or 
inconsistent items for principals proceeded in two stages: logical imputation based on other items 
reported for the principal, following defined rules, and hot deck imputation that assigned responses 
from other principals with similar characteristics. Specific details are provided in chapter VIII of 



the 1990-91 Round Data File User's Manual (Gruber, Rohr, and Fondelier, 1993) and in SASS 
Specifications Memoranda. Imputation flags were included on final data tapes. 

In Round 3, as in Round 2, questionnaire items that should have had answers but did not, had 
values imputed for them. Where possible, values were deduced from other questionnaire data 
(stage 1 methods); otherwise they were imputed by hot deck imiputation methods, with the value for 
a missing response being taken from a responding case with similar characteristics (stage 2 
methods). All but a few exceptional cases of imputation were performed at stages 1 and 2 using 
computer processing. Manual (stage 3) methods occurred when there was no responding donor for 
hot deck imputation, when the imputed value was outside its acceptable range, or when there were 
only a few unanswered cases for an item. BIA School Principal Survey questionnaires were 
manually imputed because of their relatively small number and because item response rates were 
generally high. Imputation flags were included on final data tapes. 

Weighting. In all rounds, the weighting procedures for the School Principal Survey were of the 
same form as those used for the School Survey (described in section 2.7.4), using overall weights 
that were the product of four factors: a basic sampling weight; a sampling adjustment factor; a 
school principal nonresponse adjustment factor; and a frame ratio adjustment factor. Since these 
surveys were fielded separately, there were some instances in which a questionnaire was obtained 
for the school principal but not for the school (and vice versa). Also, as noted above, some schools 
had no principals. For these reasons, weighting was done separately for the School Survey and the 
School Principal Survey. 

As discussed in section 3.8, an analysis of response rates by state or association, urbanicity, school 
level, and school size showed some response rate differences across subgroups. All of these 
classification variables were used in some way for nonresponse adjustments in order to reduce the 
potentially biasing effects that this differential nonresponse could have on survey estimates. 

Variance estimation. The computation of sampling errors for the School Principal Survey follows 
the same procedures that were used for the School Survey (see section 2.7). In the first and second 
rounds, a balanced half-sample replication (BHR) variance procedure (also known as balanced 
repeated replication (BRR)) was used to estimate sampling errors. Replicate weights for use in 
such estimates of sampling error are included on all SASS public-use microdata files. In Round I 
the replicates and replicate weights were developed independently for schools and school principal, 
but using the same general rules (Kaufman, 1991). In Round 2 the replicates for school principals 
were the same as those used for their schools (Kaufman and Huang, 1993). In Round 3, unlike the 
previous two rounds, the BRR variance estimation method was replaced by a bootstrap variance 
estimation method (see section 2.7). 

The estimated coefficients of variation (CVs) for selected estimates from Round 3 of the School 
Principal Survey are shown in table 3-10 for public school principals and table 3-11 for private 
school principals (where the CV is the standard error of the estimate divided by the estimate). The 
CVs for the percent of public school principals whose highest earned degree is a master's degree 
are less than 10 percent for all states except Connecticut (11.2 percent), Georgia (17.2 percent), 
Kentucky (11.3 percent), and Minnesota (15.6 percent). In these cases, the large CV can be 
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Table 3-10. — Selected estimates and coefficients of variation (CVs) for public school principals, by 
state: Round 3 



Percent whose 
highest degree earned 
State was master's degree 


CV 

(percent) 


Percent who worked as a 
department head before 
becoming a principal 


CV 

(percent) 


Average total 
earned income 


CV 

(percent) 


All public school principals 


63.4 


1.0 


19.6 


2.7 


$54,858 


0.2 


Alabama 


42.7 


7.9 


10.7 


20.5 


45,554 


1.0 


Alaska 


69.1 


4.1 


16.5 


11.3 


65,982 


1.4 


Arizona 


66.5 


5.9 


20.5 


19.1 


54,092 


1.1 


Arkansas i 


72.4 


4.1 


9.5 


23.6 


41,797 


1.1 


California 


68.2 


4.6 


22.1 


10.5 


62,499 


0.8 


Colorado 


62.3 


7.7 


28.5 


12.2 


52,585 


1.1 


Connecticut 


16.6* 


11.2 


16.4 


14.6 


76,803 


0.7 


Delaware 


73.9 


4.7 


28.0 


12.8 


63,921 


1.0 


District of Columbia 


76.2 


5.4 


44.5 


10.7 


66,616 


0.7 


Florida 


70.9 


4.7 


42.0 


8.0 


57,684 


0.9 


Georgia 


13.7* 


17.2 


21.9 


10.3 


54,763 


0.7 


Hawaii 


50.9 


8.3 


64.7 


8.7 


53,425 


0.7 


Idaho 


67.1 


4.8 


15.5 


14.1 


45,293 


1.3 


Illinois 


68.1 


3.8 


20.1 


9.7 


56,099 


1.0 


Indiana 


54.9 


6.6 


17.8 


16.0 


54,325 


0.7 


Iowa 


75.4 


4.9 


15.7 


22.4 


48,524 


1.2 


Kansas 


70.2 


4.4 


14.4 


14.1 


49,932 


1.2 


Kentucky 


40.3 


11.3 


14.1 


20.1 


52,279 


0.9 


Louisiana 


66.3 


4.4 


25.3 


9.8 


43,237 


0.8 


Maine 


68.6 


6.7 


14.1 


16.7 


46,769 


1.4 


Maryland 


78.3 


3.2 


30.2 


9.2 


64,258 


0.8 


Massachusetts 


66.6 


4.5 


14.9 


12.6 


56,960 


0.7 


Michigan 


61.9 


6.3 


20.1 


17.6 


62,516 


1.1 


Minnesota 


19.8* 


15.6 


18.8 


17.0 


55,500 


1.4 


Mississippi 


58.2 


6.8 


17.3 


14.7 


40,930 


0.7 


Missouri 


53.0 


8.9 


10.4 


19.4 


47,529 


1.6 


Montana 


86.7 


3.2 


19.2 


16.4 


42,382 


1.0 


Nebraska 


59.1 


7.9 


18.7 


20.1 


45,569 


2.8 


Nevada 


69.8 


5.2 


24.4 


11.7 


60,677 


0.6 


New Hampshire 


69.3 


6.4 


19.5 


15.1 


51,193 


1.2 


New Jersey 


72.9 


5.7 


15.1 


18.7 


75,863 


1.1 


New Mexico 


73.7 


5.3 


23.3 


17.0 


42,068 


0.9 


New York 


40.8 


9.7 


22.4 


12.1 


69,936 


1.6 


North Carolina 


46.0 


8.3 


27.3 


12.4 


50,548 


1.1 


North Dakota 


59.1 


6.2 


10.2 


19.4 


36,095 


2.0 


Ohio 


80.8 


3.9 


18.4 


16.3 


$53,409 


1.4 


Oklahoma 


71.6 


4.2 


14.1 


13.0 


41,599 


0.7 


Oregon 


61.5 


7.5 


24.8 


14.0 


51,798 


1.7 


Pennsylvania 


63.7 


7.6 


14.7 


20.5 


60,995 


1.2 


Rhode Island 


73.1 


6.3 


, 11.4 


22.4 


56,608 


0.6 


South Carolina 


53.9 


8.2 


19.8 


15.1 


50,805 


0.8 


South Dakota 


82.2 


2.7 


10.6 


15.8 


37,063 


1.0 


Tennessee 


67.2 


6.9 


10.9 


19.3 


44,774 


1.4 


Texas 


75.6 


3.1 


20.2 


14.2 


49,205 


0.8 


Utah 


55.4 


4.8 


22.4 


9.3 


47,920 


0.9 


Vermont 


63.5 


7.5 


14.7 


17.6 


49,234 


2.7 


Virginia 


76.7 


5.1 


18.0 


20.0 


54,801 


1.3 


Washington 


75.7 


5.0 


19.9 


15.0 


60,782 


1.0 


West Virginia 


84.0 


4.1 


6.8 


26.5 


44,091 


0.9 


Wisconsin 


68.1 


6.0 


17.9 


20.0 


54,956 


1.1 


Wyoming 


68.8 


3.9 


24.5 


15.3 


47,649 


1.2 



♦Percents are small in these states because the majority of principals have a degree higher than a master's degree. 
SOURCE: CVs were computed using estimates and standard errors from Bobbitt, Broughman, and Gruber (1995). 



Table 3-11. — Selected estimates and coefficients of variation (CVs) for private school principals, 
by association group: Round 3 



Association group 


Percent whose 
highest degree 
earned was 
master’s degree 


CV 

(percent) 


Percent who 
worked as a 
department head 
before becoming 
a principal 


CV 

(percent) 


Average total 
earned income 


CV 

(percent) 


All private school principals 


59.0 


1.1 


18.7 


4.4 


$22,739 


0.9 


Catholic 


63.6 


1.1 


23.5 


5.2 


22,356 


0.8 


Parochial 


70.6 


1.2 


18.6 


8.6 


20,058 


0.9 


Diocesan 


62.1 


2.2 


27.4 


8.7 


22,720 


1.1 


Private order 


45.0 


3.5 


44.2 


8.0 


28,426 


1.8 


Other religious 


57.8 


1.9 


14.5 


9.5 


20,440 


2.1 


Conservative Christian 


66.3 


2.6 


18.8 


13.1 


16,839 


2.2 


Affiliated 


55.0 


2.9 


15.0 


11.3 


23,995 


1.6 


Unaffiliated 


50.8 


4.8 


8.0 


20.0 


20,471 


5.8 


Nonsectarian 


52.9 


2.7 


20.2 


10.2 


26,712 


1.5 


Regular 


52.2 


3.2 


25.8 


11.6 


26,768 


2.0 


Special emphasis 


54.5 


3.9 


11.1 


22.7 


25,431 


2.7 


Special education 


53.7 


7.8 


22.2 


27.3 


27,763 


3.4 



SOURCE: CVs were computed using estimates and standard errors from Bobbitt, Broughman, and Gruber (1995). 



explained by the low percent of principals whose highest earned degree is a master's degree 
(generally less than 20 percent as compared with the national average for public school principals 
of 63.4 percent). The CVs for the percent of public school principals who worked as department 
heads before becoming principals are considerably larger: all but a few exceed 10 percent and 
about one in five exceeds 20 percent. These CVs are higher than those for the master's degree item 
because of the lower percent of principals involved: overall, only 19.6 percent of public school 
principals had been department heads before becoming principals as compared with 63.4 percent 
whose highest earned degree was a master's degree. The estimates of average total earned income 
for public school principals by state are very precise,, with only three of the states having CVs of 2 
percent or more. 

Table 3-11 displays similar findings for private school principals. The CVs for the percent of 
private school principals whose highest degree is a master's degree are less than 10 percent for all 
association groups. The CVs for the percent of private school principals who worked as 
department heads before becoming principals are considerably larger, and the CVs for average total 
earned income are small for all association groups. 



3.10 EVALUATION OF ESTIMATES 



Few alternative data sources can be used in comparisons with estimates obtained from the School 
Principal Survey. Prior to the publication of the 1987-88 Round 1 SASS estimates, the survey 
estimates of numbers of public and private school principals were compared with principal and 
school counts from other sources, including the 1985-86 Private School Survey (PSS), the 1987-88 
Common Core of Data (CCD), the Quality Education Data File (QED), which served as the frame 
for the School Survey, and an estimate of the number of public school principals from a list 
compiled by a commercial market data firm (Hammer, 1989b). Differences among the estimates 
were relatively small and were deemed to be accounted for by differences in definition and time 
reference among the estimates examined. Differences between SASS estimates of number of 
schools and number of principals in both sectors were accounted for primarily by the existence of 
schools with no principals. 

In an attempt to check on the quality of estimates of public school principals' salaries, state 
education agencies in several states were asked to provide independent information on average 
principals' salaries in their states (Hammer, 1989a). Four states — Alabama, Illinois, Kansas, and 
Maryland — provided information but, in general, the salaries provided either were not directly 
comparable with the survey estimates or there was not enough supporting documentation to 
determine the extent of comparability. 

Also for Round 1, a pre-publication review of estimates based on a set of items about hours spent 
by principals on school related activities led to a recommendation, which was followed, that these 
estimates not be included in publications (Hammer, 1990). Three factors were cited as possibly 
leading to under-reporting of hours: there was no imputation for individual items for which there 
was no response (see section 3.9.2); there was no "other" category in which to report hours not 
covered by the named activity categories; and the set of items did not ask for any distinction 
between time spent during school hours and time spent after school hours. 

The Round 2 estimates for school principals were compared with corresponding data from Round 
1. No unusual differences were noted (Hammer, 1992). For Round 3, similar comparisons showed 
no deviations. 
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4. THE TEACHER DEMAND AND SHORTAGE SURVEY 

4.1 INTRODUCTION 

The Teacher Demand and Shortage Survey (TDSS), first conducted in 1983-84, is the oldest of 
the group of surveys that was joined together to form the Schools and Staffing Surveys (SASS). 
The purpose of the TDSS is to collect data that can be used to measure the supply and demand 
for elementary and secondary school teachers, to identify teacher categories, types of schools, 
and types of school districts for which shortages exist, and to examine policies that may 
influence teacher supply and demand (salary, benefits, retirement plans, and incentive plans for 
hiring). The survey also collects some information on library media specialists/librarians. 
Information for public schools and for BIA schools that are part of an LEA is obtained directly 
from the school district (LEA). In the case of private schools and BIA schools not part of an 
LEA, the information is collected from the school. 

This chapter does not repeat the design features and procedures already described in chapter 2. 
Like the previous chapter, if has sections covering the main phases of the survey operations: 
sampling frames (4.2), sample design (4.3), content (4.4), data collection procedures (4.5), data 
processing (4.6), measurement error (4.7), nonresponse (4.8), estimation (4.9), and evaluation of 
estimates (4.10). Most of the material in this chapter concerns the public sector component of 
the TDSS. The main features of the private sector and BIA components, especially for Rounds 2 
and 3, have been described in chapter 2 in connection with the School Survey. 

4.2 SAMPLING FRAMES 

The target population for the public school sector for all three rounds of the TDSS (Round 1 in 
1987-88, Round 2 in 1990-91, Round 3 in 1993-94) consisted of U.S. public school districts, 
often called LEAs. An LEA is a local government agency administratively responsible for 
providing public elementary and/or secondary instruction and educational support services, 
operating under a public board of education. A few LEAs do not operate schools but hire 
teachers for schools in other LEAs; an example would be a special education program whose 
teachers are placed in regular schools. Such LEAs were included in the target population if they 
employed teachers in elementary and/or secondary schools. LEAs that did not hire any teachers 
were excluded. Public schools operated outside the local public school system, such as those 
operated by the BIA and the U.S. Department of Defense (DOD), were each treated as a separate 
I£A. Thus, BIA schools not found on the CCD file are considered separate entities, whereas 
BIA schools on the CCD file are considered part of the public school district (LEA) in which 
they are located. For DOD schools, if there was more than one school on a base, then only one 
TDSS questionnaire was sent to the base. In Minnesota and Missouri, a small number of schools 
operated by state agencies (e.g., state operated school for the blind) were excluded in error from 
the LEA target population in Round 2, although they were included in the first and third rounds. 

The target population for the private school sector consisted of all private schools eligible for the 
School Survey, as described in section 2.2. The sampling frame, sample design and selection 
procedures for the private school sample are described in chapter 2. Similarly the samples of 
Indian schools were obtained by the procedures described in chapter 2. 



The sample of LEAs that operate public schools is obtained by including in the TDSS all LEAs 
associated with one or more schools sampled for the School Survey. Thus, the sampling frame 
for those LEAs is the frame used for selecting public schools, the Quality Education Data (QED) 
for Round 1 and the CCD for Rounds 2 and 3 (see section 2.4.1). 

A relatively small number of LEAs that do not operate public schools but do hire teachers were 
included in the sampling frame at each round. There were 1,077 such LEAs on the frame in 
Round 1, 1,352 in Round 2, and 651 in Round 3. The decrease in this type of LEA in Round 3 
was due to the closure of some and the consolidation of several such LEAs into a single LEA. 
Almost all LEAs without schools were declared out of scope because they did not hire teachers. 

Frame evaluation. As already noted in section 2,4.3, in Round 1 a comparison of public school 
estimates with counts from the Public Elementary/Secondary School Universe Survey of the 
CCD series of surveys showed that 275 LEAs with only elementary schools had not been 
included in the school frame based on the QED list for Round 1. As a result, the numbers of 
LEAs at the national level and especially for Nebraska were underestimated from the sample for 
that round. These schools, and hence the corresponding LEAs, were included in the frame for 
later rounds. 

As part of the postprocessing edit in Round 2, weighted estimates of the counts of LEAs by state 
were compared with the 1988-89 CCD, which served as the sampling frame for Round 2, and the 
weighted numbers of teachers and students reported by the sampled LEAs were compared with 
the corresponding numbers from the 1990-91 CCD, covering the same reference year. Eight 
states had estimates of public school teachers or students from the TDSS that exceeded the CCD 
counts by 15 percent or more. Examination showed that these estimates were overestimates 
caused by the erroneous inclusion in the TDSS of LEAs that were supervisory unions or other 
districts that did not hire teachers. All such districts were reclassified as out of scope and their 
data were eliminated from the estimates. 

Estimates of LEAs and full-time equivalent (ETE) teachers from the survey compared to frame 
totals and external sources in Round 3 (see section 4.10) identified no frame deficiencies. LEA 
estimates by state were compared to the numbers on the 1991-92 CCD file. Explanations for the 
discrepancies observed included low response rates to the TDSS and possible LEA closings that 
may have occurred between the sampling year and the year of data collection (Gruber, Rohr, and 
Fondelier, 1996). TDSS estimates of the nurhbers of FTE teachers per state were compared to 
counts from CCD's State Nonfiscal Survey data (Johnson, 1995). Possible reasons for 
discrepancies did not include frame deficiencies. 

4.3 SAMPLE DESIGN 

The sample design for selecting public schools is described in section 2.4.1. In all three rounds, 
the TDSS included all LEAs associated with one or more of the sampled public schools. This 
design gave every LEA that operated schools a calculable non-zero probability of selection 
(equal to one minus the probability that none of its schools was selected for the School Survey). 

However, an exception to this basic approach occurred in Delaware, Nevada, and West Virginia. 
A simulation study prior to Round 1 showed that the sampling errors of LEA estimates in those 
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three states would be large even though the proposed selection procedures would have included 
most of the LEAs in all three states in the sample (Kaufman, 1991, p.27). In all rounds, 
therefore, every LEA in those three states was treated as a separate stratum for the purpose of 
sampling schools, so that all of the states' LEAs were included in the sample. 

The sample of LEAs that did not operate schools but that hired teachers was obtained by 
selecting a sample of all LEAs that did not operate schools and eliminating those that did not hire 
teachers at the data collection stage. LEAs that did not operate schools were sampled differently 
in each round. In Round 1, those that reported having teachers were Sampled with probability 
proportional to the square root of the number of teachers. Those LEAs with teacher information 
unknown were sampled with equal probability at a rate of about 1/20 from a list ordered by state, 
urbanicity, percent minority, first three digits of ZIP Code and LEA identification (ED). In 
Rounds 2 and 3, all UEAs without schools were selected using equal probability systematic 
sampling at the rates of 1 in 10 and 1 in 6, respectively. In Round 2, in order to give the benefits 
of implicit stratification, the list was ordered before sampling by metropolitan area status, three- 
digit ZIP Code, and LEA ID. In Round 3, the sort variables for these I£As were LEA type code 
(indicating who operates the agency), state, number of teachers, and UEA ID. The sampling of 
LEAs that did not operate schools but hired teachers was eliminated for Round 4 since it added 
more to the variance than it contributed to the scope. 

Table 4-1 displays the sizes of the LEA samples selected for the TDSS for the first three rounds. 
The total eligible sample declined slightly over the rounds, dropping from about 5,600 LEAs in 
Round 1 to under 5,400 in Round 3. A substantial proportion of sampled UEAs that did not 
operate schools were found to be ineligible. LEAs were classified as ineligible (out of scope) if 
the district no longer existed; the district served only prekindergarten and/or kindergarten 
students;: the district served only postsecondary or adult education students; or the agency was 
not a school district or other public agency that employed elementary and/or secondary teachers. 
Information on the sample size for private schools can be found in section 2.4.2. 

Table 4-1. — Local education agency (I^A) sample sizes: Rounds 1-3 



Category 


Round 1 
(1987-88) 


Round 2 
(1990-91) 


Round 3 
(1993-94) 


Total eligible sample 


5,594 


5,394 


5,355 


LEAs with schools 


5,586 


5,380 


5,350 


LEAs without schools 


70 


135 


109 


Eligible LEAs without schools 


8 


14 


5 



SOURCES: Kaufman (1991); NCES (1991); Jabine (1994); Gruber, Rohr, and FondeUer (1993, 1996). 

4.4 CONTENT 

In all three rounds, the TDSS collected data: 



■ To measure the supply and demand for elementary and secondary school teachers; 
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■ To identify teacher categories and types of schools and schopl districts for which shortages 
exist; 

■ To examine policies that may influence teacher supply and demand (salary, benefits, 
retirement plans, and incentive plans for hiring); and 

■ In Rounds 2 and 3, to estimate the number of teachers who held certification in their field of 
assignment. 

Boe (1996) has noted the TDSS is the only source of sorhe teacher data, such as district teacher 
counts, hiring criteria, and collective bargaining agreements. 

Rounds 2 and 3 included as a final question "Not counting interruptions, how long did it take to 
complete this survey?" For mail responses, this item was completed by the person responding 
for the LEA; for followup telephone iriteiwiews, the item was completed by the interviewer. In 
Round 2, the median time for completion was 1 hour and 15 minutes, with an interquartile range 
of 90 minutes. About 5 percent of the LEAs required more than 5 hours to complete the 
questionnaire and 1 percent required more than 10 hours.. 

Content changes. There were only a few changes in survey content between Rounds 1 and 2. To 
reduce the burden on respondents, a complex accounting-style matrix ite:m that called for data on 
FTE teachers and teaching positions by level and specialty, was dropped. (See also the State Data 
Project in section 4.7). Conversely, items were added in Round 2 to collect information on 
demand for and shortages of librarians and on pension portability. 

Items in the Round 2 questionnaire asking about the. previous year's enrollment, the number of 
postsecondary students and teachers, teacher benefits, and merit pay for teachers were deleted 
from the Round 3 questionnaire. Nonetheless, the Round 3 questionnaire contained five sections 
compared to the two sections of the previous round (enrollment and teaching positions, and 
district policies), reflecting significant , additions in content. The five sections in the Round 3 
questionnaire were as follows: , 

■ Enrollment inforrnation — counts of students by grade level and race/ethnicity, number of 
days in the school year, and data on the release of results from st^dardized tests. 

■ Teacher informaition — FTE counts of teachers employed by the LEA, counts of certified, 
itinerant, newly hired, and laid off teachers; vacant and abolished teaching positions; counts 
of teachers by race/ethnicity; LEA criteria for considering teaching applicants; and the 
number of newly hired teachers with emergency certification. 

■ Library media specialists/librarians information — FlE counts of librarians employed by the 
LEA, vacant librarian positions, abolished librarian positions, and librarians laid off at the 
end of the last school year. 

■ Programs and services — ^This new section included information on prekindergarten 
programs. Chapter 1 services, participation in the National School Lunch program, and 
enrollment choice programs. 



■ District policies — high school graduation requirements, drug abuse prevention and discipline 
policies, teacher salaries, teacher retirement plans, teacher incentive plans, and staff training 
programs. 

The additional items reflected both recommendations included in various research reports and 
the need to supplement information gaps. Six of the questions added in Round 3 were pretested 
in the 1991-92 field test. Six other sets of questions included in the field test were not included 
in the Round 3 TDSS because of the following: they had low response rates, they were unclear 
and could lead to inconsistent or invalid data, they created undue response burden, or changed 
education research priorities. 

4.5 DATA COLLECTION PROCEDURES 

Questionnaires. In all three rounds, the TDSS used separate questionnaires for the sampled 
LEAs in the public sector and the sampled private schools in the private sector. In Round 1, the 
private school version was sent to the private schools sampled for the School Survey as a 
separate questionnaire. For Rounds 2 and 3, the TDSS and the School Survey questions for 
private schools were combined into a single questionnaire. A combined questionnaire was also 
used for Indian schools in Rounds 2 and 3. 

Pretests. The questionnaires for Round I of the TDSS were field tested on 186 LEAs and 75 
private schools (Nash, n.d.). Further in-depth interviews were conducted with 45 randomly 
selected respondents to the TDSS pretest to determine how the data were derived and what was 
included and excluded from reported counts. These in-depth interviews, which were conducted 
by telephone by professional staff members from both the U.S. Census Bureau and the National 
Center for Education Statistics (NCES), resulted in several changes being made to the 
questionnaire. These changes involved adding "head counts" to the instruction when asking for 
student enrollment; adding three new teaching fields in the elementary and secondary FTE 
positions by field matrix; adding an instruction to report bilingual teachers only in the bilingual 
category in this same matrix; allowing LEAs the option of reporting contracts in days or in 
months; and eliminating an item on partial retirement age/service requirements. 

In the Round 2 pretest, questionnaires were mailed to 332 LEAs during the 1989-90 school year. 
Some potential reporting problems were identified (Healy, 1990a). One dealt with the categories 
used for grade level in questions about staffing and enrollment: prekindergarten, kindergarten, I- 
6, and 7-12. Some districts use other grade structures for their school data — -kindergarten to 5, 6- 
8, and 9-12, for example. In the pretest, 12 of 283 LEAs handled this problem by crossing out 
the grade level categories on their questionnaires and writing in new ones. This observation 
suggested the need for a clerical check of completed questionnaires to make adjustments in such 
cases. 

Numerous instances of incomplete or incorrect reporting in the pretest were also observed with 
the matrix items that called for data on FTE teachers and positions by grade level and specialty. 
As noted earlier, these two items were dropped from the final questionnaire for Round 2. 

A general observation from this pretest review was that the quality of questionnaires returned by 
mail appeared to surpass that of the ones that had been completed by telephone followups. This 
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may be because the questionnaire was difficult to complete by telephone and also because some 
of the telephone cases were "quasi refusals," with respondents reluctantly providing minimal 
data. 

A pretest for Round 3 of SASS was conducted during the first half of 1992 with U.S. Census 
Bureau field representatives collecting the data by telephone. Debriefing comments from Census 
regional offices relating specifically to the public TDSS questionnaire dealt primarily with issues 
of respondent burden resulting from the complexity of the questionnaire (difficulty with 
breakdowns of counts into categories) and the requirements to complete matrix-style items (too 
much time required to complete such questions). Understanding of the FTE concept continued 
to be considered a problem and was reported as the most difficult and time-consuming area for 
respondents to understand and for interviewers to work with over the telephone; one U.S. Census 
Bureau interviewer was quoted as saying, "I have yet to find one respondent who I feel really 
understands the concept of FTE" (U.S. Census Bureau, 1992). The debriefing of the 
interviewers also indicated that the district administrators were the most difficult of all SASS 
respondents to contact and were the least cooperative. Furthermore, T.F.A s felt that they had too 
many similar surveys from the state and that the information being requested could be found 
elsewhere (1993-94 SASS pretest memorandum). While the CCD and other surveys at the state 
level probably do collect the same or similar information about counts of teachers and students, 
these sources have not been available for incorporation into the TDSS datafile (see the State Data 
Project in section 4.7). The items asking about FTE teachers were reduced in Round 3 and have 
been virtually eliminated in Round 4. 

Procedures. In Rounds 2 and 3, the TDSS was essentially part of the School Survey for private 
schools and Indian schools. In Round 1, although a separate questionnaire was used for the 
private TDSS, the same data collection procedures used for the School Survey applied (see 
section 2.5). 

Data collection activities for LEAs were generally the same as those described for schools in 
earlier chapters. Generally there was an advance mailing in late summer/early fall to T.F.A s and 
school principals to explain the nature of the SASS data collection activities. This was followed 
by a mailing of the TDSS questionnaires in late fall or early winter. A second mailing of 
questionnaires was sent to LEAs that had not yet responded by 5 or 6 weeks after the initial 
mailing, with telephone followup occurring later. However, there were a few notable differences 
between this and other SASS surveys. 

In Rounds 2 and 3, unlike the first round, the advance letter mailed to LEA superintendents to 
introduce SASS also indicated that a U.S. Census Bureau representative would be calling to ask 
that they designate a staff member to take responsibility for completing the TDSS, thus allowing 
the TDSS to be the only SASS instrument addressed to a named person (Monaco, Salvucci, 
Zhang, Hu, and Gruber, 1998). If there was a designated staff member, the questionnaire was 
sent to that person. Otherwise, the questionnaire was mailed to the superintendent. Any 
knowledgeable LEA employee was allowed to respond to the questionnaire. For some LEAs, the 
data were provided by several respondents. In Round 3, in order to obtain data consistent with 
the CCD frame data, the range of grades for which information was desired also was clearly 
labeled. As for other Round 3 components of SASS, a toll-free U.S. Census Bureau phone 



number was included on the questionnaire for use by respondents in need of assistance — in 
particular, if the LEA grade range was different from that shown on the questionnaire label. 

In all three rounds, trained U.S. Census Bureau interviewers contacted mail nonrespondents by 
telephone in an attempt to complete the questionnaires. Paper-and-pencil instruments were used 
for all rounds. CATI was not used for followup in Round 3 because of time and budget 
constraints but is being used in Round 4. In Round 3, about 90 percent of sampled LEAs 
responded by mail. With the telephone followup cases, if the person contacted by telephone was 
unwilling to provide the information over the telephone, the interviewer tried to persuade the 
person to complete the questionnaire and return it by mail. Of 'the 590 followup cases, 574 (97 
percent) were completed. 

Quality assurance. As was done for all of the SASS suiweys at each round, U.S. Census Bureau 
regional office staff reviewed a sample of the questionnaires completed by interviewers in their 
telephone followups of nonrespondents. The procedures were the same as those described for 
the School Survey in section 2.6.1. 

4.6 DATA PROCESSING 

The sequence and nature of the data processing procedures for all three rounds of the TDSS were 
essentially the same as for the other basic SASS surveys (see the processing procedures for the 
School Survey described in section 2.7.) In summary, these procedures included assigning a 
status code to each questionnaire to indicate its interview status, a general clerical edit to review 
items for legibility, grouping of questionnaires for data entry, data entry with 100 percent 
independent verification, assigning the preliminary interview status, computer editing (pre-edit 
and edit) that checked ranges and within-item and between-item data consistency, making 
corrections where deterministically possible, and assigning a final interview status code. 

Editing. In Round 2, the most frequent reason for pre-edit rejection of LEA questionnaires was 
that the number of students or teachers was at least 35 percent greater than expected from the 
CCD. In Round 3, almost 20 percent of the sample LEA records were rejected for further review 
during the pre-edit based on II criteria. The three criteria that caused rejection most frequently 
were the following: 

s The number of teachers was at least 25 percent greater than expected (10 percent); 

a The number of students was at least 20 percent greater than expected (4 percent); 

a The number of teachers was at least 20 percent less than expected (4 percent). 

For rejected LEA cases, in all three rounds, U.S. Census Bureau staff reviewed the computer 
record and the questionnaire. When possible, unacceptable entries were corrected by using 
information on the questionnaire or sample frame information. Respondents were sometimes 
contacted to resolve problems in Rounds I. and 2 but not in Round 3. Incorrect entries that could 
not be corrected were deleted from the record. 
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Despite the sizable percent of pre-edit rejections in Round 3, few changes were made to the LEA 
responses based on these rejections. Where changes were made, they consisted of correcting 
inconsistencies or deleting unacceptable responses that could not be corrected. Of the 165 
variables in the pre-edit, only 15 variables had change rates greater than 1 percent and only 5 had 
change rates greater than 2 percent, where the change rate is the number of changes made to an 
item divided by the number of completed questionnaires. The 5 questions with change rates 
greater than 2 percent were the following: a question that asked for FTE teachers in the district 
for the grades 1-12 (4 percent); a question that asked for the total number of FTE teachers in the 
district (3 percent); a question that asked for the number of white, non-Ehspanic teachers in the 
district (3 percent); a question that asked how many of the FTE teachers reported had 
standard/regular state certifications in their fields of assignment (3 percent); and a question that 
asked for the total number of students enrolled (2 percent). 

After pre-edit corrections were made the data file was submitted to a computer edit that consisted 
of a range check, a consistency edit, and a blanking edit. Of the 166 variables in the Round 3 
computer edit, change rates ranged from a low of 0.1 percent to about 46 percent; for 136 of the 
variables in the computer edit, the change rate was less than 10 percent. For the most part, the 
variables with the highest change rates were grade indicator variables that were to be checked off 
if the district offered the specified grade. Another question with a very high change rate asked 
for the high and low ends of the range of teachers' yearly base salaries in the district. This 
question was asked only if the LEA could not answer a prior question asking for more detailed 
information about the district's salary schedule. 

4.7 MEASUREMENT ERROR 

No reinterview studies were conducted for the TDSS in any of the three rounds. Frequently, 
more than one person in an LEA provided data, which could have caused difficulties in arranging 
for repeat data collections, especially if conducted by telephone. Therefore, unlike some of the 
other SASS surveys, there is no information on measurement errors available from this source. 
There are, however, a variety of studies that shed light on measurement error in the TDSS. 

Teacher and student counts in Round 1. For Round 1, at the time the weights were being applied 
to the LEA sample records, it was observed that for some LEAs the numbers of students or 
teachers were much higher or lower than expected on the basis of prior year data from the CCD 
for the same LEAs. A listing was prepared of all LEAs for which (a) reported counts of students 
or teachers differed by 35 percent or more from expected counts, or (b) the student/teacher ratio 
was greater than 35 or less than 10. This listing contained 290 LEAs for which counts of 
students or teachers appeared to be incorrect, including 46 LEAs that appeared to have reported 
data for sample schools only, rather than the entire district. 

Further review of the questionnaires for the 290 LEAs revealed several cases where LEAs had 
merged and one case that was out of scope because all teachers and students were preschool. 
There were also 33 LEAs for which the entire questionnaire had been completed for a single 
school, rather than the entire LEA. (This is the converse of the situation discussed in section 2.8 
where it was found that some of Round 1 School Survey questionnaires had been completed for 
the entire LEA rather than the specified sample school.) These 33 sample LEAs were 
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reclassified as noninterview cases, reducing the response rate for the public TDSS by slightly 
more than 1 percent. 

As a result of these findings and the actions taken, the sample weights had to be recalculated. 
For the LEAs that were not reclassified as noriinterviews, values that were clearly incorrect were 
replaced by imputed values, based primarily on other iteiris from the same questionnaire or data 
on the sample file (Fondelier, 1990). As described above, in Rounds 2 and 3, checks of LEA 
student and teacher counts were incorporated into the pre-edit procedures. 

The State Data Project. In Round 1, many LEA respondents to the TDSS had difficulty 
completing two matrix items that called for detailed information on FTE teachers and positions 
by teaching level and specialty. Some of the state education offices suggested that they might be 
in a better position than the LEAs to report such data to NCES. A feasibility test of this 
approach, called the State Data Project, was included in the 1990 SASS Pretest for Round 2 
(Healy, 1990b). 

Initially, 1 1 states were selected to participate in the test. For these states, data for several items 
on the TDSS questionnaire were to be collected directly from the sample LEAs and also, 
independently and in computer-readable form, from the state education offices. Data from the 
two sources would be compared, on an item-by-item basis, for each of the sample LEAs. 

Eventually, 7 state education offices were able to submit data tapes with the requested LEA data. 
Pretest interviews were completed for 82 of the 96 LEAs in the pretest sample for those states. 
For 24 of the 38 data elements compared, over 50 percent of the individual LEAs reported data 
that differed by 10 percent or more from the data provided by their state offices. On the basis of 
these results, NCES concluded that it would not be advisable to try to obtain data for sample 
LEAs from a centralized state source in Round 2 of SASS. It was also decided, as noted earlier, 
that the matrix items on FTE teachers and positions would not be included in the survey 
questionnaire for Round 2. 

This outcome does not necessarily rule out the collection of some of the LEA data for the TDSS 
from state education offices in future rounds of SASS. In November 1991, NCES and the 
Council of Chief State School Officers convened a Workshop on Improving Reliability and 
Comparability of Staffing Data, the main purpose of which was to review the findings from the 
State Data Project. The participants concluded that state education agencies can report LEA data 
for selected items. The state participants expressed a desire to continue to work with NCES to 
develop a state reporting role in SASS, especially for the public sector component of the TDSS 
(Blank, 1992). However, the results of the State Data Project demonstrated that further research 
would be needed to better understand the nature of the discrepancies that were observed and to 
identify the specific items that could be adequately reported at the state level. 

Measurement errors in Round 2. Indications of measurement error in Round 2 of the public 
TDSS came from several sources: an early review of unedited and edited questionnaires received 
from the field (Healy and Pasqualucci, 1991); memoranda submitted by census regional offices 
following the completion of SASS data collection for that round (U.S. Census Bureau, 1991a); a 
review of the forms completed by regional office staff for their reviews of questionnaires 
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obtained by U.S. Census Bureau interviewers in telephone followups of mail nonrespondents 
(Pasqualucci, 1991); and a review of pre-edit reject rates, edit change tallies, and post-edit item 
response rates (Jenkins, 1992a). 

Types of measurement errors mentioned in these, sources included the following: 

B As noted in connection with the 1990 pretest, a few respondents changed the grade level 
categories for reporting enrollments and staff, for example from 1-6 and 7-12 to 1-8 and 9- 
12. This was observed in the early review of incoming questionnaires and a procedure was 
added to the clerical edit to check for this type of alteration. 

o As in Round 1, many respondents failed to record decimal entries in the manner intended for 
the items relating to FIE staff and graduation requirements. In most instances, such 
reporting errors were detected and coirected either in clerical edits or as a result of 
consistency checks included in computer edits. 

a Many respondents failed to observe skip instructions and unnecessarily completed an item on 
the overall range of base year teacher salaries in the district. This item was intended only for 
respondents who could not provide separate ranges for different levels of qualification. 
Failure to skip did not affect the accuracy of the data for these items. 

H There were frequent discrepancies between component items and overall totals for two 
topics, one relating to FTE teachers and one to FTE librarians and media specialists. 

In most instances, it was possible to detect errors of these kinds in the clerical and computer edits 
and to substitute correct or at least more nearly correct values on the basis of other information 
on the questionnaire or in the sample file, or from telephone contacts in the early stages of 
processing. 

4.8 NONRESPONSE 

Unit nonresponse. Since unit response rates relate only to the eligible population, sampled LEAs 
and private schools that were not operating in the school year of reference for the survey or 
failed to meet the definition for other reasons are excluded from consideration. In both Rounds 2 
and 3, the percent of the initial sample of LEAs that was excluded for such reasons was relatively 
small, at around 4 percent. As noted above, in Round 1 a few questionnaires that were initially 
counted as interviews were excluded subsequently from the survey estimates because of 
reporting errors that could not readily be corrected; these cases were counted as nonrespondents 
if they were eligible for the survey. 

Table 4-2 shows weighted unit response rates by state for LEAs for each of the rounds, and table 
4-3 shows the corresponding rates for private schools. (In Rounds 2 and 3, the private TDSS 
was combined with the School Survey questionnaire, so that the response rates for these rounds 
are the same as those in table 2-13 for the School Survey.) The Round 1 weighted response rate 
for LEAs was about 25 percentage points higher than that for the private school TDSS 
questionnaire (91 percent vs. 66 percent). For Round 2, the weighted LEA response rate was 
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Table 4-2. — TDSS weighted response rates for public local education agencies, by state: 
Rounds 1-3 



Weighted response rate (percent)* Weighted response rate (percent)* 



State 


Round 1 
(1987-88) 


Round 2 
(1990-91) 


Round 3 
(1993-94) 


State 


Round 1 
(1987-88) 


Round 2 
(1990-91) 


Round 3 
(1993-94) 


Total 


91 


94 


94 










Alabama 


97 


96 


94 


Montana 


89 


95 


94 


Alaska 


100 


96 


94 


Nebraska 


92 


97 


97 


Arizona 


92 


90 


99 


Nevada 


100 


100 


100 


Arkansas 


96 


91 


98 


New Hampshire 


84 


93 


87 


California 


90 


91 


91 


New Jersey 


84 


86 


77 


Colorado 


96 


98 


89 


New Mexico 


88 


95 


98 


Connecticut 


61 


77 


94 


New York 


91 


96 


94 


Delaware 


95 


100 


90 


North Carolina 


88 


94 


97 


District of Columbia 


100 


100 


100 


North Dakota 


94 


94 


96 


Florida 


93 


92 


99 


Ohio 


98 


89 


100 


Georgia 


81 


92 


98 


Oklahoma 


98 


99 


94 


Hawaii 


100 


100 


100 


Oregon 


99 


91 


98 


Idaho 


97 


96 


94 


Pennsylvania 


84 


94 


90 


Illinois 


93 


92 


93 


Rhode Island 


100 


92 


100 


Indiana 


98 


96 


89 


South Carolina 


84 


93 


94 


Iowa 


90 


98 


92 


South Dakota 


97 


98 


96 


Kansas 


85 


99 


94 


Tennessee 


92 


100 


97 


Kentucky 


87 


92 


99 


Texas 


90 


95 


97 


Louisiana 


92 


90 


89 


Utah 


97 


96 


96 


Maine 


88 


92 


96 


Vermont 


99 


87 


99 


Maryland 


88 


88 


83 


Virginia 


91 


91 


88 


Massachusetts 


84 


94 


97 


Washington 


81 


97 


98 


Michigan 


97 


90 


97 


West Virginia 


87 


98 


96 


Minnesota 


87 


92 


90 


Wisconsin 


85 


96 


91 


Mississippi 


93 


97 


98 


Wyoming 


93 


96 


85 


Missouri 


92 


94 


98 











*The basic weight is used to reflect the probability of selection. 
SOURCES: NCES (1991c); Gruber et al. (1993, 1996), 
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Table 4-3. — ^TDSS weighted response rates for private schools, by association group: 
Rounds 1-3 





Weighted response rate (percent)* 




Round 1 


Round 2^ 


Round 3^ 


Association group 


(1987-88) 


(1990-91) 


(1993-94) 


Total, area frame and list frame 


70 


84 


83 


Area sample 


49 


74 


71 


Association list frame 


— 


— ■ 


84 


Association of Military Colleges and Schools of U.S. 


92 


91 


96 


National Catholic Education Association, 


84 


91 


89 


Jesuit Secondary Education Association 


Friends Council on Education 


78 


91 


80 


National Association of Episcopal Schools 


66 


89 


78 


National Society of Hebrew Day Schools 




71 


79 


Solomon Schechter Day Schools 


► 53 


85 


88 


Other Jewish ^ 




70 


72 


Lutheran Church - Missouri Synod 




96 


91 


Evangelical Lutheran Church - Wisconsin 




98 


90 


► 83 






Evangelical Lutheran Church in America 




96 


95 


Other Lutheran 




94 


88 


General Conference of Seventh Day Adventists 


72 


94 


89 


Christian Schools International 


89 


94 


69 


American Association of Christian Schools 


39 


59 


73 


National Association of Private Schools for Exceptional Children 


65 


87 


89 


American Montessori Society Schools, other Montessori 


74 


86 


83 


National Association of Independent Schools 


64 


85 


82 


National Independent Private School Association 


i . 


t 


80 


Other private schools 


55 


81 


79 



available. (Data were not collected or not reported.) 

^Not applicable. (Not listed as a separate association group.) 

'The basic weight is used to reflect the probability of selection. 

^Rates for Rounds 2 and 3 are rates for private schools since the TDSS questions were included in the private school questionnaire for these 
rounds. 

SOURCES: NCES (1991c); Gruber et al. (1993, 1996). 

about 94 percent, compared with 84 percent for the private School Survey questionnaire (which 
included the teacher demand and shortage items). Thus, the gap between the public and private 
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school sectors narrowed considerably. For both sectors, the response rates increased between 
Rounds 1 and 2, despite an anticipated decline because of an overlap in schools across rounds. 
In fact, in Round 2 the response rate for overlap LEAs (those that had been in the sample in 
Round 1) was about the same as the rate for nonoverlap LEAs. In Round 3, both the weighted 
TDSS response rates and the gap between public and private schools remained about the same as 
in Round 2. 

The higher response rates in later rounds may have been due, in part, to the elimination of the 
troublesome matrix items on FTE teachers and positions. Additionally, combining the School 
Survey and TDSS questionnaires for private schools probably contributed to the higher response 
rate for such schools. For private schools in Round 1, when separate questionnaires were used, 
the response to the School Survey questionnaire was higher than the response to the TDSS 
questionnaire (79 percent vs. 66 percent). 

Table 4-2 shows the considerable variation in weighted response rates by state within the public 
sector. In Round 1, 18 states had weighted LEA response rates of 95 percent or better and only 
1, Connecticut, had a rate as low as 61 percent. In Round 2, 7 more states than in Round 1 had 
weighted response rates of 95 percent or better and, as before, Connecticut had the lowest rate 
(77 percent). Round 3 saw 27 states achieve a response rate of 95 percent or more, and only 1 
state. New Jersey, recorded a response rate below 80 percent; Connecticut, which had the lowest 
response rate in the previous two rounds, achieved a rate of 94 percent. 

As pointed out in chapter 2, in assessing the variation in response rates among the association 
groups in table 4-3, account needs to be taken of the small sample sizes for many of the 
association groups. The table shows that the improvement in response rates that occurred 
between Round 1 and the later rounds applied across all association groups and also for the area 
frame component. However, the rates in 7 groups dropped by at least 5 percentage points 
between Rounds 2 and 3, including a drop of 25 percent for the Christian Schools International 
Association Group. In all rounds, the response rates for the area frame cornponent were lower 
than those for the list frame. 

As with the other SASS components, in Rounds 2 and 3 studies were carried out to examine 
possible variation in response rates to the TDSS across different types of LEAs (Scheuren, et al., 
1996; Monaco, Salvucci, Zhang, Hu, and Gruber, 1998). (The corresponding analyses for 
private schools are reported in chapter 2 since the items for the TDSS were incorporated into the 
School Survey questionnaire for these rounds.) Some results from these studies are presented in 
table 4-4. 

Significance tests of differences in response rates across the categories of the characteristics 
displayed in table 4-4 were conducted for only four characteristics — number of schools in the 
LEA, student enrollment, urbanicity, and region — in Round 2. These tests were multiple 
comparison tests of differences between categories, using a Bonferroni adjustment and a 10 
percent significance level. Significant differences between at least one pair of categories were 
found for three of the four ch^acteristics. Significance tests were applied to all six 
characteristics in Round 3, using an overall .significance- test (Rao-Scott chi-squared statistic). 
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Table 4-4. — ^Weighted response rates for local education agencies, by selected characteristics: 
Rounds 2 and 3 





Weighted response rate (percent)^ 


Characteristics 


Round 2 (1990-91) 


Round 3 (1993-94) 


Number of schools: 


0-5 


94 


95 


6+ 


93 


91 




■ NS" 


P<0.001 


Student enrollment: 


0-299 


92 


95 


300-599 


95 


95 


600-999 


94 


95 


1000-2499 


95 


95 


2500-4999 


94 


92 


5000-9999 


93 


90 


10000-24999 


94 


91 


25000+ 


89 


89 




S" 


P=0.001 


Urbanicity: 


Outside of MSA 


94 


95 


MSA not central city 


92 


93 


MSA central city 


92 


93 




S" 


P=0.037 


Region: 


West 


93 


94 


South 


95 


96 


Northeast 


91 


91 


Midwest 


94 


95 




S" 


P<0.001 


Large LEA“: 


Yes 


t 


89 


No 




94 

P<0.001 


LEA sampled in 1990-91: 


Yes 




94 


No 


t ■ 


94 

P=0.774 



*Not applicable. 

^The basic weight is used to reflect the probability of selection. 

^NS indicates there are no significant differences at Ae 10 percent significance level. 

indicates a significant difference between the response rates for at least some of the levels of the variable at the 10 percent significance level, 
large LEA is a LEA that was sampled with certainty and also that nonresponse for the sampled unit would have a major effect upon its state 
estimates. 

SOURCES: Scheuren et al. (1996); Monaco et al. (1998). 

The tests showed significant differences at a 5 percent significance level across categories for all 
the characteristics with the exception of whether or not the LEA had been sampled in 1990-91. 
The pattern of significance was the same for Rounds 2 and 3, with the exception of the 
characteristic "Number of schools," for which there was no significant difference in Round 2 but 
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a highly significant difference in Round 3. Most of the category differences in response rates are 
not large. In both survey rounds, higher response rates were found for LEAs outside of an MSA, 
and lower response rates were found for LEAs in the Northeast and for LEAs with large student 
enrollments. In Round 3, LEAs classified as large had lower response rates than those not 
classified as large. 

Item nonresponse. Table 4-5 summarizes item response rates for all three survey rounds of the 
TDSS. The minimum item response rate was 40 percent for the public TDSS in Round 1; 12 
percent of the items had response rates less than 75 percent, and 74 percent had response rates of 
at least 90 percent. For the private sector component, the minimum item response rate was 16 
percent, 18 percent of items had rates of less than 75 percent, and 70 percent of items had 
response rates of 90 percent or more. Most of the items with low response rates in Round 1 were 
associated with the two matrix items on FTE teachers and positions by field of assignment. 
Because of the low response rates and other indications of poor quality, no data based on these 
two questionnaire items were published and they were not included in the survey data files. 



Table 4-5.— Percent of items with selected response rates for the Teacher Demand and Shortage 
Survey, by sector: Rounds 1-3 





Percent of items with response rate: 


Minimum of item 


Sector 


> 90 percent 


<75 percent 


response rate (percent) 


Round 1 (1987-88) 


Public 


74 


12 


40 


Private 


70 


18 


16 


Round 2 (1990-91) 


Public 


90 


0 


85 


Private 


— 


— 


— 


Round 3 (1993-94) 


Public 


91 


1 


67 


Private 


— 


— 


— 



“Not available. (For the private sector in Rounds 2 and 3, the TDSS questions were included on the private school questionnaire. The 
unweighted item response rates in table 2-15 include the rates for the private sector TDSS questions.) 

SOURCES: NCES (1991c); Gruber. Rohr, and Fondelier (1993. 1996). 



In Round 2, which did not include these items, the overall level of item response was 
substantially improved. For LEAs, the item response rates varied between 85 and 100 percent, 
with 90 percent of items having response rates of 90 percent or better. No items had response 
rates less than 75 percent for the LEAs. 

In Round 3, apart from one item, the LEA item response rates ranged from 83 percent to 100 
percent. The exception was an item that asked whether the high school graduation requirements 
reflected a 3- or 4-year program; it had a response rate of 67 percent. Ninety-one percent of the 
items had response rates greater than 90 percent, with the majority of those above 95 percent. 
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4.9 ESTIMATION 



Imputation. Items that were missing or failed range or consistency checks were imputed at 
various stages of processing in the same manner in Rounds 1 and 2. In the first stage of 
computer imputation, missing values were imputed using information about the LEA from other 
items on the same questionnaire, from data for the LEA from the CCD frame and, if all of the 
schools in an LEA had been included in the sample, from the School Survey questionnaires for 
those schools. For items that could not be completed in the first stage, a hot deck procedure 
based on responses for other LEAs with similar characteristics was used at the second stage. 
Imputation flags were assigned to imputed values. 

In Round 3, imputation consisted of three stages of which the first two were the same as in the 
previous rounds. As part of the stage 1 imputation, ratio adjustments were applied to two sets of 
questions: enrollment and numbers of part-time and full-time teachers. For records where the 
sum of "students by race/ethnicity" entries did not equal the LEA's total enrollment, the "students 
by race/ethnicity" entries were ratio adjusted so that their sum would equal the total enrollment 
value. The "teachers by race/ethnicity" entries were similarly handled. For the "students by 
race/ethnicity" question that consisted of five subquestion items, ratio adjustments were made for 
between 22 and 30 percent of the subquestion entries. For the "teachers by race/ethnicity" 
question, between 5 and 10 percent of the five subquestion items were ratio adjusted. Other 
stage 1 methods assigned values for very small percents of values. Stage 2 hot deck imputation 
methods assigned values for from less than 1 percent of responses for several variables, to as 
high as 33 percent for a question asking whether the high school graduation requirements 
reflected a 3 or 4 year program. For most items, the extent of imputation for this stage ranged 
between 1 and 6 percent. The third stage of imputation consisted of clerical imputation for an 
item that had fewer than 10 missing responses and for one or two cases for some other items. 

Imputation flags were included on the data file in Round 3 as in previous rounds, but the range of 
flag values was extended to indicate the type of imputation, if applicable (ratio adjustment, 
another type of stage 1 imputation, stage 2 imputation, clerical imputation). In earlier rounds, 
the flags distinguished only between imputed and nonimputed values. 

Weighting. Because of the method used to select the sample of LEAs, calculation of the basic 
sampling weights for LEAs was somewhat more complex than it was for schools, school 
principals, and teachers. Three types of LEA need to be distinguished: 

■ All LEAs in Delaware, Nevada, and West Virginia were selected with certainty in all three 
rounds. Hence their basic weight is 1. 

■ LEAs that operated a school or schools were selected if one or more of their schools were 
sampled for the School Survey. The schools were sampled from a stratified design with six 
strata. The basic weight assigned to a sampled LEA that operated schools was therefore the 
inverse of one minus the product of the probabilities that no school operated by the LEA was 
selected from any of the six strata (see Kaufman (1991), Kaufman and Huang (1993), and 
Abramson et al. (1996) and section 2.4.1). 
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■ LEAs that did not operate schools were sampled at the rate of 1 in 10 in Round 2 and 1 in 6 
in Round 3. Hence their basic weights were 10 and 6 in Rounds 2 and 3 respectively. In 
Round 1, such LEAs were divided into two sets: those with teacher counts available on the 
frame were sampled with probabilities proportional to the square root of their numbers of 
teachers, and those without teacher counts were sampled at a rate of 1 in 20. Sampled LEAs 
from the former set were assigned basic weights that were inversely proportional to the 
square roots of their numbers of teachers. Those in the latter set were assigned basic weights 
of 20. 

The LEA basic weights were adjusted by similar adjustments to those described for schools in 
section 2.7.4. These were a sampling adjustment factor, a noninterview factor, and a ratio 
adjustment factor that benchmarks the estimated numbers of LEAs to known frame totals. The 
sampling adjustment factor adjusts for unusual circumstances affecting the LEA's probability of 
selection, such as a merger with another LEA or an LEA split or duplicate listings of the LEA in 
the sampling frame. The definitions of adjustment cells and the collapsing rules for the 
nonresponse and frame ratio adjustment factors were similar to those used in the School Survey. 
When assessing the collapsing of cells for frame ratio adjustments in Round 3, it was found that 
collapsing had a large impact on the estimates for California, Pennsylvania, and Maine. 
Collapsing criteria were modified in these cases to reduce the impact on the final estimates. 

Variance estimation. As described in section 2.7, a balanced half-sample replication (BHR) 
procedure was used to estimate sampling errors for all SASS surveys in both Rounds 1 and 2. 
To reflect the fact that LEAs were selected through the school sample, the LEA replicates were 
formed using the corresponding school replicates. An LEA was placed into an LEA replicate if 
any of the sample schools associated with the LEA had been included in the corresponding 
school replicate. LEAs that had been selected with certainty were included in all replicates, and 
a separate procedure was used to assign sample LEAs with no schools to replicates. See 
Kaufman (1991), Kaufman and Huang (1993), and Abramson et al. (1996) for more detail. 

In Round 1, sample schools from responding noncertainty LEAs were assigned to a variance 
stratum/half-sample using the school procedure. If there was only one sample school in an LEA 
and the LEA contains only one school, the LEA received the school's set of 48 replicate weights. 
If there was more than one school, the LEA fell into a given replicate if any of its schools fell 
into the replicate. Replicate weights for LEAs were recalculated using the procedures described 
in the weighting section, with the exception that the probability of the LEA being selected in a 
given school stratum was divided by two. Certainty LEAs had replicate weights of 1 and were 
placed in all replicates. Replicate weights for use in estimating sampling errors by this procedure 
are included on all microdata files, so that users of these files can estimate sampling errors for 
items of interest to them. 

As in Round 1, the majority of LEAs in Round 2 were selected through the sample of schools, 
and the 48 half-sample replicates for these LEAs were formed using the corresponding school 
replicates, following the procedures described for Round 1. A separate procedure was used to 
assign sample LEAs with no schools to replicates. LEAs without schools were sorted by order of 
selection. Pairs of LEAs were then systematically placed into consecutive variance strata and 
each element of a variance stratum was assigned to alternating half-samples. After variance strata 
were assigned, an orthogonal matrix was used to form the 48 replicates. 
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As mentioned in previous chapters, the BHR variance estimation procedure assumes sampling 
with replacement, whereas sampling without replacement is used for all of the SASS surveys. 
Violation of the replacement assumption leads BHR to overestimate the true variances, but the 
effects should be small unless the sampling fractions are quite large (Kaufman, 1991). However, 
for some states, the proportion of LEAs sampled is large, so substantial overestimates of variance 
are likely to occur for the public TDSS estimates for these states. 

The public TDSS is different from the other SASS surveys in that its unit for data collection and 
analysis, the LEA, is an aggregate of the sampling units, which are schools. This sample design 
leads to possible violation of a second assumption that is implicit in the BHR method of variance 
estimation, namely that the true variance is inversely proportional to the sample size. Kaufman 
(1992, 1993, 1994) undertook a series of simulation experiments to determine the extent of bias 
for the BHR variance estimation procedures and to evaluate some alternatives. One finding was 
that the BHR variance estimates for the public TDSS for Rounds 1 and 2 were substantial 
overestimates. The extent of overestimation varied by state; for some states confidence intervals 
based on estimates plus or minus one standard error covered the corresponding population values 
more than 90 percent of the time (if the variance estimates were unbiased, this should happen 
about 68 percent of the time). 

One of the simulation experiments evaluated two different weighting procedures: the one used, 
in which the sampling weight for each LEA is the inverse of its selection probability, and an 
alternative "expected hits" weighting procedure, in which the weights are based on the selection 
probabilities of the sample schools within the selected LEAs. The alternative weighting 
procedure satisfies the second of the two assumptions of the BHR method of variance estimation 
and, therefore, should produce unbiased estimates of variances, using the BHR method, if 
sampling were done with replacement. This proved to be the case. However, for averages and 
ratios, the estimates based on expected hit weights had larger variances than those based on 
probability weights, so a change in the weighting scheme was not deemed advisable. 

The same study also evaluated two different methods of variance estimation; the BHR and the 
bootstrap method. Using the bootstrap approach did not affect the ability of users to compute 
variances using any BHR program without modification (Kaufman, 1993, 1994). Accordingly, 
for Round 3 SASS surveys, including the TDSS, a bootstrap variance estimator was 
implemented. Bootstrap replicate weights for LEAs were developed as they were for schools. 

Tables 4-6 and 4-7 present coefficients of variation (CVs) for selected estimates from Round 3 of 
the TDSS. The CV is computed as the standard error of an estimate divided by the estimate, and 
the standard error is computed using the bootstrap method described above. The CVs for the 
percent of an LEA's FTE teacher count who are continuing teachers and newly hired teachers are 
all small, 0.5 percent or less for continuing teachers and 7 percent or less for newly hired 
teachers (the latter CVs being larger because of the small percent of newly hired teachers). 
Table 4-7 presents estimates and CVs for teacher salary schedules based on education and 
experience. Almost all CVs are below 2 percent. 
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Table 4-6. — Coefficients of variation (CVs) for the percents of full-time equivalent (FTE) public 
school teachers that are continuing and for newly hired public school teachers, by 
state: Rounds 



States 


Continuing FTE 
teachers (percent) 


CV 

(percent) 


Newly hired FTE 
teachers (percent) 


CV 

(percent) 


50 states and D.C. 


92.1 


0.0 


7.9 


0.5 


Alabama 


91.9 


0.1 


8.1 


1.2 


Alaska 


90.7 


0.1 


9.3 


1.4 


Arizona 


88.9 


0.3 


11.1 


2.7 


Arkansas 


91.9 


0.2 


8.1 


2.5 


California 


92.0 


0.2 


8.0 


2.4 


Colorado 


90.6 


0.3 


9.4 


2.6 


Connecticut 


94.8 


0.2 


5.2 


2.9 


Delaware 


93.5 


0.0 


6.5 


0.5 


District of Columbia 


93.9 


0.0 


6.1 


0.0 


Florida 


90.3 


0.0 


9.7 


0.3 


Georgia 


90.0 


0.1 


10.0 


1.3 


Hawaii 


88.5 


0.0 


11.5 


0.0 


Idaho 


91.3 


0.1 


8.7 


1.5 


Illinois 


91.4 


0.4 


8.6 


4.1 


Indiana 


94.6 


0.2 


* 5.4 


3.0 


Iowa 


93.6 


0.2 


6.4 


3.3 


Kansas 


91.2 


0.3 


8.8 


3.1 


Kentucky 


93.2 


0.1 


6.8 


2.0 


Louisiana 


92.1 


0.1 


7.9 


0.9 


Maine 


93.6 


0.3 


6.4 


4.4 


Maryland 


91.6 


0.0 


8.4 


3.6 


Massachusetts 


93.5 


0.2 


6.5 


2.8 


Michigan 


96.4 


0.2 


3.6 


6.7 


Minnesota 


91.0 


0.4 


9.0 


4.1 


Mississippi 


89.8 


0.2 


10.2 


2.0 


Missouri 


91.2 


0.4 


8.8 


3.6 


Montana 


91.1 


0.5 


8.9 


5.4 


Nebraska 


92.9 


0.5 


7.1 


7.0 


Nevada 


89.6 


0.0 


10.4 


0.1 


New Hampshire 


92.2 


0.3 


7.8 


4.0 


New Jersey 


94.5 


0.2 


5.5 


4.2 


New Mexico 


87.5 


0.1 


12.5 


0.7 


New York 


95.5 


0.2 


4.5 


3.1 


North Carolina 


89.2 


0.2 


10.8 


1.4 


North Dakota 


93.3 


0.4 


6.7 


5.1 


Ohio 


94.6 


0.2 


5.4 


3.9 


Oklahoma 


90.9 


0.3 


9.1 


2.9 


Oregon 


94.8 


0.2 


5.2 


3.7 


Pennsylvania 


92.8 


0.3 


7.2 


4.3 


Rhode Island 


97.1 


0.04 


2.9 


0.7 


South Carolina 


90.9 


0.1 


9.1 


1.4 


South Dakota 


92.1 


0.2 


7.9 


2.0 


Tennessee 


91.7 


0.1 


8.3 


1.3 


Texas 


88.0 


0.2 


12.0 


1.6 


Utah 


92.0 


0.03 


8.0 


0.4 


Vermont 


93.8 


0.5 


6.2 


6.8 


Virginia 


91.1 


0.1 


8.9 


1.5 


Washington 


91.6 


0.2 


8.4 


2.0 


West Virginia 


97.6 


0.0 


2.5 


0.0 


Wisconsin 


96.1 


0.2 


4.9 


3.1 


Wyoming 


91.6 


0.1 


8.4 


1.1 



SOURCE: CVs were computed using estimates and standard errors from Bobbitt, Broughman, and Gruber (1995). 
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Table 4-7. — Coefficients of variation (CVs) for average public school teacher salary schedules, 
by earned degree and experience, by state: Round 3 



States 


Bachelor's and 
no experience 


CV 

(percent) 


Master's plus 30 
credits & no 
experience 


CV 

(percent) 


Highest step or 
salary schedule 


CV 

(percent) 


50 states and D.C. 


$21,923 


0.3 


$25,201 


0.3 


$40,517 


0.3 


Alabama 


22,263 


0.2 


27,061 


0.3 


32,840 


0.5 


Alaska 


31,374 


0.6 


38,165 


0.5 


58,095 


0.5 


Arizona 


21,890 


1.3 


25,938 


1.3 


40,661 


1.2 


Arkansas 


19,603 


0.7 


22,046 


0.7 


29,685 


0.9 


California 


24,404 


2.0 


28,126 


1.3 


46,272 


1.5 


Colorado 


19,937 


0.8 


23,491 


1.4 


37,316 


2.7 


Connecticut 


28,195 


1.0 


32,379 


0.9 


56,189 


1.7 


Delaware 


22,914 


0.1 


28,254 


0.1 


47,743 


0.2 


District of Columbia 


22,000 


0.0 


34,000 


0.0 


54,000 


0.0 


Florida 


21,838 


0.3 


24,394 


0.4 


39,599 


0.5 


Georgia 


20,065 


0.3 


23,899 


0.6 


42,134 


0.4 


Hawaii 


25,436 


0.0 


27,352 


0.0 


49,199 


0.0 


Idaho 


18,102 


0.4 


22,225 


0.6 


33,128 


1.1 


Ulinois 


21,415 


1.4 


26,056 


1.2 


42,004 


2.0 


Indiana 


22,560 


0.6 


25,100 


1.1 


41,993 


0.6 


Iowa 


18,796 


0.5 


22,225 


1.0 


33,317 


0.8 


Kansas 


22,714 


0.6 


26,056 


0.6 


36,671 


0.7 


Kentucky 


21,135 


0.3 


26,733 


0.4 


36,743 


0.4 


Louisiana 


18,045 


0.7 


18,468 


0.7 


30,539 


0.5 


Maine 


19,566 


0.8 


21,997 


1.1 


• 36,814 


0.9 


Maryland 


24,833 


0.2 


28,240 


0.4 


48,158 


0.4 


Massachusetts 


23,108 


0.7 


26,718 


0.6 


44,783 


0.7 


Michigan 


24,705 


1.0 


28,571 


1.3 


48,315 


1.5 


Minnesota 


21,965 


0.5 


26,101 


1.0 


38,638 


1.0 


Mississippi 


19,008 


0.1 


20,534 


0.3 


32,693 


0.2 


Missouri 


18,158 


0.7 


20,779 


1.0 


28,222 


1.8 


Montana 


17,801 


0.4 


21,466 


1.2 


33,755 


1.1 


Nebraska 


17,781 


0.7 


23,063 


1.5 


32,281 


1.0 


Nevada 


24,220 


0.1 


28,910 


0.1 


44,958 


0.2 


New Hampshire 


21,317 


0.8 


25,125 


1.7 


38,971 


1.4 


New Jersey 


28,424 


0.8 


32,396 


1.5 


58,208 


1.4 


New Mexico 


22,114 


0.1 


23,833 


0.7 


35,994 


1.3 


New York 


27,441 


0.9 


31,869 


1.3 


59,116 


1.5 


North Carolina 


20,077 


0.1 


22,409 


0.7 


38,733 


0.5 


North Dakota 


16,624 


0.5 


19,444 


0.9 


27,371 


1.0 


Ohio 


20,550 


0.6 


23,832 


0.8 


42,152 


1.0 


Oklahoma 


22,158 


0.2 


23,690 


0.5 


30,445 


0.5 


Oregon 


20,708 


0.6 


23,719 


0.9 


35,962 


2.0 


Pennsylvania 


26,341 


,1.3 


29,333 


1.6 


50,337 


1.7 


Rhode Island 


23,423 


0.1 


25,701 


0.2 


46,016 


0.2 


South Carolina 


20,354 


0.5 


25,375 


0.5 


41,766 


0.6 


South Dakota 


17,895 


0.3 


20,131 


0.7 


27,617 


0.8 


Tennessee 


21,348 


0.6 


24,754 


0.9 


34,650 


1.6 


Texas 


19,011 


0.6 


19,264 


0.6 


32,358 


0.5 


Utah 


18,740 


0.2 


21,176 


0.4 


34,900 


0.6 


Vermont 


20,918 


0.7 


25,399 


0.9 


40,330 


1.1 


Virginia 


23,098 


0.8 


25,074 


0.9 


38,328 


1.3 


Washington 


21,441 


0.1 


26,216 


0.5 


44,892 


0.1 


West Virginia 


21,466 


0.0 


26,024 


0.0 


36,678 


0.0 


Wisconsin 


23,080 


0.3 


27,936 


0.8 


42,995 


0.8 


Wyoming 


20,137 


0.3 


24,775 


0.3 


38,701 


0.3 



SOURCE: CVs were computed using estimates and standard errors from Bobbitt, Broughman, and Gruber (1995). 
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4.10 EVALUATION OF ESTIMATES 



As noted previously, some school districts in Nebraska that had only elementary schools were 
found to have been omitted from the Round 1 sampling frame based on the QED file. 
Consequently, estimated numbers of LEAs, schools, teachers, and students for Nebraska were 
low in Round 1. 

The final Round 2 estimated numbers of LEAs from SASS were compared with two counts from 
the 1988-89 CCD by state: the total number of LEAs and the number of regular LEAs. These 
comparisons were complicated by the fact that the character and definitions of LEAs vary by 
state; some of the nonregular LEAs in the CCD had teachers and were eligible to be included in 
SASS. For 14 states, the SASS estimate of LEAs differed from the CCD count of regular or total 
LEAs by 15 percent or more. Estimates for these states were reviewed in detail and in some 
instances state agencies or LEAs were called to obtain information about the nature of 
nonregular LEAs. For each of the 14 states, a CCD count of LEAs was determined that came as 
close as possible to meeting the SASS definition for eligible LEAs, and the SASS estimate was 
compared with that count. Based on this comparison, the SASS estimate was within 10 percent 
of the CCD count in 10 of the states and within 15 percent in the remaining 4 states. 

Nationally, the SASS estimate of public school teachers, as reported by the LEAs, was 5.9 
percent below the CCD count in Round 2. There were 4 states whose SASS estimates were more 
than 15 percent below the CCD counts; Maryland, Michigan, New Mexico, and Texas. The 
largest underestimate was for New Mexico, which was 18.6 percent below the CCD count. For 
enrollment, the U.S. estimate from SASS was 2.7 percent below the CCD count, and there were 
3 states — Michigan, Nevada and New Mexico — for which SASS estimates of enrollment were 
from 10 to 15 percent below the CCD counts. For all other states, SASS estimates were within 
10 percent of the CCD figures. Details by state for all of these comparisons are provided in 
chapter XII of Round 2 Data File User's Manual (Gruber et al., 1993). 

In Round 3, SASS estimates of the numbers of LEAs by state were compared to the numbers on 
1991-92 CCD file, which was the sampling frame for Round 3. Since not all LEAs were 
considered to be in-scope for SASS, three comparisons were made — one to the total number of 
LEAs in each state (table 4-8), one to the number of regular LEAs in the state (table 4-8) and one 
to the adjusted sampling frame (developed at the U.S. Census Bureau in preparation for SASS), 
which provides a number between the total number of LEAs and the number of regular LEAs 
(table 4-9). The SASS counts of LEAs were generally closer to the adjusted sampling frame 
counts, except for Minnesota and Nebraska, where the adjusted sample seemed to make little or 
no difference. Upon investigation, it was determined that Minnesota had a large number of 
administrative LEAs, some of which hire teachers. Thus, the adjusted sample number of LEAs 
is 18 higher than the number of regular LEAs published by CCD. However, the SASS sample 
underestimated the regular, adjusted sample, and total number of LEAs by 12 percent, 17 
percent, and 26 percent, respectively. Several possible explanations were offered. One such 
explanation was that since the sample of LEAs is dependent upon the sample selected for schools 
(i.e., schools are the main unit of analysis and districts are associated with the sampled schools); 
there may be a number of LEAs that are not being represented very well in the overall Minnesota 
district sample. LEAs included in the frame that do not operate schools are supposed to be 



Table 4-8. — Public school districts in SASS, by state, compared with total and regular districts in 
1991-92 Public Education Agency Universe (Common Core of Data [CCD]): 

Round 3 



State 


CCD 
total districts 


CCD 

regular districts 


SASS 

districts 


SASS/CCD 
total districts 
(percent) 


SASS/CCD 
regular districts 
(percent) 


50 states and D,C, 


16,661 


15,173 


14,998 


90 


99 


Alabama 


133 


129 


130 


98 


101 


Alaska 


56 


56 


56 


100 


100 


Arizona 


247 


229 


219 


89 


96 


Arkansas 


347 


325 


311 


90 


96 


California 


1,070 


1,005 


1,062 


99 


106 


Colorado 


196 


176 


183 


93 


104 


Connecticut 


179 


166 


177 


99 


107 


Delaware 


22 


19 


19 


86 


100 


District of Columbia 


1 


1 


1 


100 


100 


Florida 


74 


69 


72 


97 


104 


Georgia 


• 183 


183 


183 


100 


100 


Hawaii 


1 


1 


1 


100 


100 


Idaho 


114 


114 


114 


100 


100 


Illinois 


1,020 


942 


933 


91 


99 


Indiana 


319 


296 


310 


97 


105 


Iowa 


469 


435 


399 


85 


92 


Kansas 


304 


304 


303 


100 


100 


Kentucky 


255 


176 


174 


68 


99 


Louisiana 


83 


66 


71 


86 


108 


Maine 


326 


283 


236 


72 


83 


Maryland 


24 


24 


24 


100 


100 


Massachusetts 


439 


352 


349 


80 


99 


Michigan 


622 


561 


608 


98 


108 


Minnesota 


517 


434 


383 


74 


88 


Mississippi 


171 


150 


154 


90 


103 


Missouri 


544 


543 


541 


99 


100 


Montana 


621 


539 


523 


84 


97 


Nebraska 


895 


777 


650 


73 


84 


Nevada 


18 


17 


18 


100 


106 


New Hampshire 


239 


174 


163 


68 


94 


New Jersey 


620 


608 


591 


95 


97 


New Mexico 


96 


88 


95 


99 


108 


New York 


758 


717 


724 


96 


101 


North Carolina 


135 


133 


123 


91 


92 


North Dakota 


325 


276 


257 


79 


93 


Ohio 


791 


613 


661 


84 


108 


Oklahoma 


593 


593 


564 


95 


95 


Oregon 


308 


296 


280 


91 


95 


Pennsylvania 


611 


503 


593 


97 


118 


Rhode Island 


38 


37 


37 


97 


100 


South Carolina 


95 


95 


94 


99 


99 


South Dakota 


227 


189 


176 


78 


93 


Tennessee 


140 


140 


139 


99 


99 


Texas 


1,051 


1,051 


1,048 


100 


100 


Utah 


47 


40 


40 


85 


100 


Vermont 


339 


279 


237 


70 


85 


Virginia 


161 


141 


143 


89 


101 


Washington 


296 


296 


296 


100 


100 


West Virginia 


55 


55 


55 


100 


100 


Wisconsin 


428 


428 


428 


100 


100 


Wyoming 


58 


49 


50 


86 


102 



NOTE: Districts that do not operate schools nor hire teachers are out of scope for SASS, although such districts may appear on the CCD frame. 
These districts, denoted "districts without schools” have been subtracted from the state’s weighted estimate of districts whenever those districts 
were inadvertently retained on the SASS file with a final weight. However, out of 20 unweighted districts without schools on the SASS file with a 
final weight, 5 are valid. The rest have been subtracted from the states weighted count 
SOURCE: Gruber, Rohr, and Fondelier (1996), 
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Table 4-9. — Public school districts in SASS, by state, compared with census-adjusted number of 
districts in the Public Education Agency Universe (Common Core of Data [CCD]) 
for 1991-92: Rounds 



State 


CCD districts 
(adjusted) 


SASS districts 


SASS/CCD 

(percent) 


50 states and D.C. 


15,472 


14,998 


97 


Alabama 


130 


130 


100 


Alaska 


56 


56 


100 


Arizona 


222 


219 


99 


Arkansas 


325 


311 


96 


California 


1,067 


1,062 


100 


Colorado 


183 


183 


100 


Connecticut 


178 


177 


99 


Delaware 


19 


19 


100 


District of Columbia 


1 


1 


100 


Rorida 


72 


72 


100 


Georgia 


183 


183 


100 


Hawaii 


1 


1 


100 


Idaho 


114 


114 


100 


Illinois 


1,011 


933 


92 


Indiana 


311 


310 


100 


Iowa 


425 


399 


94 


Kansas 


304 


303 


100 


Kentucky 


176 


174 


99 


Louisiana 


74 


71 


96 


Maine 


238 


236 


99 


Maryland 


24 


24 


100 


Massachusetts 


358 


349 


97 


Michigan 


615 


608 


99 


Minnesota 


462 


383 


83 


Mississippi 


163 


154 


94 


Missouri 


541 


541 


100 


Montana 


533 


523 


98 


Nebraska 


771 


650 


84 


Nevada 


18 


18 


100 


New Hampshire 


163 


163 


100 


New Jersey 


590 


591 


100 


New Mexico 


94 


95 


101 


New York 


727 


724 


100 


North Carolina 


135 


123 


91 


North Dakota 


277 


257 


93 


Ohio 


661 


661 


100 


Oklahoma 


578 


564 


98 


Oregon 


295 


280 


95 


Pennsylvania 


600 


593 


99 


Rhode Island 


38 


37 


97 


South Carolina 


95 


94 


99 


South Dakota 


176 


176 


100 


Tennessee 


139 


139 


100 


Texas 


1,050 


1,048 


100 


Utah 


40 


40 


100 


Vermont 


251 


237 


94 


Virginia 


155 


143 


92 


Washington 


296 


296 


100 


West Virginia 


55 


55 


100 


Wisconsin 


428 


428 


100 


Wyoming 


54 


50 


93 



SOURCE: Gruber, Rohr, and Fondelier (1996). 
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sampled at a 10 percent rate, but the particular sample selected in a given year may have fewer 
than average in-scope LEAs (those that hire teachers), which would keep the estimated number 
of LEAs lower. Another possible explanation was that Minnesota's response rate was more than 
4 percent lower than the national average, and adjustments may not have compensated 
adequately (Gruber et al., 1996). 

Nebraska had a large number of LEAs that do not operate schools, so the valid comparison is to 
the number of regular public school districts. The adjusted sampling frame count is only 6 
districts more than the number of regular districts. In the 1990-91 SASS, there was only a 6 
percent difference between the estimated number of districts and the number of regular districts 
in the CCD. In Round 3, there was a 17 percent difference between the SASS estimated count 
and the number of regular CCD districts — 1 1 percentage points higher than the, same comparison 
in the previous round. Possible reasons cited for this included Nebraska's higher-than-average 
out-of-scope rate that reduces the number of eligible districts from the selected sample, and the 
number of district closings that may have occurred between the sampling year and the year of 
data collection may be increasing (Nebraska has a large number of one-school districts). 
Adjustments may not have been able to compensate for such occurrences (Gruber et al., 1996). 

Another comparison made in Round 3 related to the number of FTE teachers per state as 
estimated from the TDSS and CCD's State Nonfiscal Survey (Johnson, 1995). Table 4-10 shows 
that the TDSS estimate of the numbers of FTE teachers was less than 1 percent lower overall 
than CCDs. Estimates for 11 states were at least 5 percent higher, with Alaska's and 
Pennsylvania's estimates being 10 percent higher. On the other hand, four states had estimates 
that were less than 90 percent of CCD's number of FTE teachers. A possible explanation for 
TDSS undercounts in some states is that large LEAs have higher nonresponse rates for which the 
nonresponse adjustments do not fully compensate. Possible explanations for TDSS overcounts 
include the possibilities that some LEAs may report headcounts rather than FTEs and some may 
report teachers covered by other districts (e.g., teachers who have been "lent out" to other 
districts) (Gruber et al., 1996). 
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Table 4-10— Full-time equivalent (FTE) teachers in SASS, by state, compared with 1993-94 
Common Core of Data (CCD): Round 3 



State 


FTE teachers 
CCD 


FTE teachers 
SASS 


SASS/CCD 

(percent) 


SO states and D.C. 


2,505,074 


2,501,112 


100 


Alabama 


43,002 


44,468 


103 


Alaska 


7,193 


7,886 


110 


Arizona 


37,493 


39,334 


105 


Arkansas 


26,014 


27,771 


107 


California 


^ . 221,779 


215,044 


97 


Colorado 


33,661 


29,004 


86 


Connecticut 


34,526 


35,142 


102 


Delaware 


6,380 


6,555 


103 


District of Columbia 


6,056 


6,056 


100 


Florida 


1 10,653 


116,299 


105 


Georgia 


75,602 


74,405 


98 


Hawaii 


10,111 


10,300 


102 


Idaho 


12,007 


12,130 


101 


Illinois 


1 10,874 


89,862 


81 


Indiana 


55,107 


56,469 


102 


Iowa 


31,616 


32,120 


102 


Kansas 


30,283 


29,345 


97 


Kentucky 


37,324 


40,285 


108 


Louisiana 


46,913 


46,398 


99 


Maine 


15,344 


16,384 


107 


Maryland 


44,171 


43,627 


99 


Massachusetts 


58,766 


59,665 


102 


Michigan 


80,267 


80,674 


101 


Minnesota 


46,956 


42,271 


90 


Mississippi 


28,376 


29,321 


103 


Missouri 


54,543 


55,093 


101 


Montana 


9,950 


10,866 


109 


Nebraska 


19,552 


16,721 


86 


Nevada 


12,579 


12,177 


97 


New Hampshire 


11,972 


11,821 


99 


New Jersey 


84,564 


84,436 


100 


New Mexico 


18,404 


18,013 


98 


New York 


179,413 


181,499 


101 


North Carolina 


69,421 


66,259 


95 


North Dakota 


7,755 


8,100 


104 


Ohio 


107,444 


107,609 


100 


Oklahoma 


39,031 


39,269 


101 


Oregon 


26,488 


23,209 


88 


Pennsylvania 


101,301 


111,711 


110 


Rhode Island 


9,823 


10,329 


105 


South Carolina 


38,620 


39,363 


102 


South Dakota 


9,557 


9,343 


98 


Tennessee 


46,066 


48,148 


105 


Texas 


224,830 


231,393 


103 


Utah 


19,053 


20,320 


107 


Vermont 


8,102 


7,641 


94 


Virginia 


70,220 


65,637 


93 


Washington 


45,524 


47,036 


103 


West Virginia 


21,029 


20,718 


99 


Wisconsin 


52,822 


56,887 


108 


Wyoming 


6,537 


6,701 


103 



SOURCE: Gruber, Rohr, and Fondelier (1996). 
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5. THE TEACHER SURVEY 



5.1 INTRODUCTION 

The Teacher Survey collects data from a sample of classroom teachers in schools sampled in the 
School Survey. Data for teachers are collected for the same school year as for the schools (i.e., 
1987-88 for Round 1 of the Schools and Staffing Surveys (SASS), 1990-91 for Round 2, 1993- 
94 for Round 3, and 1999-00 for Round 4). Sample schools in the School Survey are asked to 
provide lists of teachers who were teaching at the school at the start of the survey reference year. 
Between 1 and 20 teachers are sampled from the list for each eligible school that provides a list, 
yielding an overall sample size in excess of 65,000 teachers at each round. Questionnaires are 
mailed to sample teachers at their schools, and nonrespondents are followed up with telephone 
interviews at school. 

The Teacher Survey collects data from teachers about current teaching status and work load, past 
teaching experience, education and training, perceptions and attitudes toward teaching, 
incentives and compensation, and demographic characteristics. The teacher questionnaire 
consists of four different forms for teachers in different types of schools: public schools, private 
schools. Bureau of Indian Affairs (BIA) schools (Rounds 3 and 4), and charter schools (Round 
4). The four versions of the questionnaires are virtually identical and include tracing information 
needed to locate teachers for the Teacher Followup Survey, which is conducted in the year after 
the Teacher Survey. Those eligible for selection for the Teacher Followup Survey include all 
sample teachers who responded to the Teacher Survey (see chapter 6). 

This chapter discusses the definition of teachers in the survey population for the Teacher Survey 
(5.2), sample design considerations (5.3), the teacher sample, including the use of teacher lists 
from sample schools in the School Survey as the sampling frame (5.4); questionnaire contents 
(5.5); data collection procedures, measurement error studies, and nonresponse (5.6); data 
processing and estimation (5.7); and evaluation of estimates (5.8). 

5.2 DEFINITION OF TEACHERS 

The target population for the SASS Teacher Survey consists of regular full-time and part-time 
teachers whose primary assignment is teaching in kindergarten or any of the grades 1 to 12 in a 
school that is in-scope for the School Survey (see section 2.2). Also included are long-term 
substitutes who are filling the role of a regular teacher on a long-term basis and itinerant teachers 
(those teaching regularly in more than one school). 

Beginning in Round 3, the definition has been extended to anyone who teaches grades K-12 
including persons whose primary assignment was not teaching. With this change, school 
principals who also teach are eligible for the teacher sample. If selected, they would receive a 
teacher questionnaire in addition to receiving a school principal questionnaire (see chapter 3). 

The survey population is determined by the operational procedures for the survey. The list frame 
of teachers within sample schools is compiled in the fall of the reference year. Teachers sampled 
from the lists are contacted later, starting in January in Rounds 1 and 2 and in December in 
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Round 3. Only teachers who remain at the schools at which they were sampled are treated as in- 
scope. Thus the survey population comprises teachers who were present at the beginning of the 
school reference year and who are still at the schools when the data collection for the Teacher 
Survey takes place. The survey population thus excludes teachers who , left between the 
compilation of the list and data collection, those who joined the schools in that period or at any 
other time in the school year, and teachers who joined newly created schools. 

In each round, schools provided information on the teacher listing forms. This information was 
used to classify the eligible teachers into the following types: 

■ New teachers — those in their first, second, and third year of teaching; all others are classified 
as experienced teachers. 

■ Bilingual/English as a Second Language (ESL) teachers — those who use a language other 
than English to instruct students with limited English proficiency; or teachers providing 
intensive instruction in English to students with limited English proficiency. 

■ Asian or Pacific Islander (API) and American Indian, Aleut, or Eskimo (AIAE) teachers — 
those indicated on the teacher listing form as API or AIAE. 

Ineligible persons. Teachers in schools ineligible for the School Survey are ineligible for the 
Teacher Survey (see section 2.2 for the definition of eligible schools). Also excluded from the 
Teacher Survey are short-term substitutes, teacher's aides, student teachers, other nonteaching 
professionals (e.g., guidance counselor, librarian, nurse, psychologist), support staff (e.g., cook, 
custodian, bus driver, dietician, secretary), and in Rounds 1 and 2, school administrators (e.g., 
principal, assistant principal). 

Persons ineligible for the Teacher Survey could be screened out at three points in the data 
collection. First, the instructions state that they should be excluded from the teacher lists 
requested from the sample schools for use in selecting the teacher sample. Second, any person 
who had been incorrectly included on a school's teacher list and sampled could be identified by 
the first item on the Teacher Survey questionnaire, which asks for the respondent's main 
assignment at the school during the school year. Respondents in categories that make them 
ineligible for the Teacher Survey (e.g., student teachers) are instructed to mail back their 
questionnaires without completing the remaining items. Finally, if a completed questionnaire is 
returned for an ineligible person, it would normally be classified as out of scope in the interview 
status edit (see section 5.7). 

5.3 SAMPLE DESIGN GOALS 

This section briefly outlines the considerations for each of the four goals that have guided the 
design of the Teacher Survey: providing estimates of acceptable precision for specified domains; 
placing limits on the number of teachers selected from each sample school; making the sample 
approximately self-weighting within each of the main domains; and ensuring that the target 
sample sizes would be achieved for bilingual, API, and AIAE teachers. 
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Domains^ The Teacher Survey is designed to provide estimates of adequate precision for the 
following domains: 

■ Experienced and new teachers separately in public elementary and secondary schools by 
state, in public combined schools, in BIA and Native American schools (since Round 2), and 
in charter schools (Round 4 only); in private schools by association group categories, school 
level, and census region; 

■ Bilingual/ESL teachers (separately for California, Texas, Florida, Dlinois, New York, and all 
other states in Round 1); and 

■ API and AIAE teachers (since Round 2). 

To provide estimates of acceptable precision for these domains requires oversampling of 
bilingual, API, AIAE teachers, and new teachers in private schools relative to other groups. At 
each round, the sample sizes of new teachers in private schools were determined to ensure that 
there would be sufficient numbers of new teachers in both the Teacher Survey and the Teacher 
Followup Survey. New teachers in public schools required no oversampling. Section 5.4 
describes the methods used to sample the teachers for the Teachers Survey. 

Placing limits on the number of teachers to sample per school. The sample design specified that 
at least one teacher be selected from every sample school. The number of teachers to be selected 
from a school was not allowed to exceed 20. This limit was imposed to avoid overburdening 
schools with a large teacher sample. 

Making the sample approximately self-weighting within each of the main analytical domains. 
This goal was accomplished by using within-school sampling fractions for selection of teachers 
that, when multiplied by the schools' selection probabilities, would produce approximately the 
same teacher selection probabilities for all schools in a domain or stratum, for example, public 
elementary schools in a state. It was expected that this design would be close to optimum with 
regard to the sampling errors of estimates for each domain. A completely self-weighting sample 
of teacher was not possible given the limits on teacher sample size per school discussed above. 

Ensuring the target sample sizes of bilingual, API, and AIAE teachers. The procedures to ensure 
the target sample sizes of bilingual, API, and AIAE were somewhat different in each round. In 
Rounds 1 and 2, there were no reliable estimates for the numbers of bilingual, API, and AIAE 
teachers in advance of sampling. Since the sampling of teachers was done on a flow basis in 
these two rounds as the lists of teachers came in from sample schools, estimates of the total 
numbers of teachers in these groups could not be made using all the teacher lists. 

To address this problem. Round 1 used initial sampling fractions expected to produce a larger 
than needed sample of bilingual teachers (API and AIAE teachers were not oversampled in this 
round). As data collection proceeded, estimates of the expected final sample sizes were 
developed and a sampling reduction rate was determined and applied to the overall sample to 
provide the desired sample sizes (see Kaufman, 1991). 
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Round 2 also used conservative sampling fractions initially to select larger than needed samples 
of bilingual, API, and AIAE teachers. The selected samples were randomly assigned into 101 
equal size "reduction groups," and then reduction groups were deleted as needed to achieve the 
target sample size for each group of teachers. 

Round 3 used a different sampling procedure that was applied in three stages. The early returns 
of teacher lists from sample schools were combined into a single list and sorted by types of 
teacher, thus providing a count of teachers by type. The sampling fraction was then determined 
to yield the desired sample size. This procedure was repeated with the next batch of returns and 
again with a final batch. 

5.4 SELECTING THE TEACHER SAMPLES 

Since the sampling procedures for teachers from schools in the public and private sectors are 
similar, they will be described together, with differences noted where they exist. This section 
discusses the collection of teacher lists from sample schools to use as the sampling frame for 
teachers (5.4.1), teacher samples and sample sizes (5.4.2), and coverage and frame quality issues 
(5.4.3). The Sample Design and Estimation reports for the three rounds (Kaufman, 1991; 
Kaufman and Huang, 1993; and Abramson et al., 1996) provide further details of the sampling 
procedures. 

5.4.1 Teacher list collection 

Teacher lists were requested from all schools sampled for the School Survey. The listing forms 
were mailed with the advance letters to schools in the fall of the survey reference year (see table 
1-2 for the mailing schedule in each round). School principals or other school staff completed 
the forms and returned them in a postage-paid envelope. The listing forms contained instructions 
for listing eligible teachers, and schools were asked to record each teacher's first and last name, 
teaching experience, race/ethnicity, participation in bilingual/ESL programs, and teaching 
specialty by level. 

In Rounds 1 and 2, U.S. Census Bureau field representatives started making telephone calls to 
sample schools that had not returned the teacher listing forms about 4 weeks after the mailing. 
They asked the schools (in order of preference) to mail in the completed listing form, to mail in a 
list of teachers, or to give the list of teachers over the telephone. If a school objected to 
providing a complete list of teachers, the school was given the option to select a sample of 
teachers as instructed by the field representative and to provide information for the selected 
teachers over the telephone. Some schools that selected their own samples objected to providing 
the names of sample teachers; these schools were asked to label the ones they had selected as Tl, 
T2, T3, etc. The questionnaires for these teachers were subsequently mailed to the contact 
person for the sample school labeled with these identifiers (U.S. Census Bureau, 1990). In 
Round 2, this procedure was used for about 1 percent of the public schools and about 3 percent 
of the private schools. This option was not used in Round 3. 

Round 3 used a slightly different data collection procedure, a new listing form, and a toll-free 
number for schools to call for assistance if necessary. One week after the first mailing, a 
reminder postcard was sent to all sample schools. About 3 weeks after the postcard, a second 
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listing form was mailed to schools that had not returned the lists (about 60 percent of the sample 
schools). Census field representatives began calling schools that had not returned the listing 
forms about 4 weeks after the second mailing. Telephone followups ended in mid-December, 
about the same time as in previous rounds. 

5.4.2 The teacher samples 

Round 1. The Round 1 teacher sample used a different selection procedure from that used in 
subsequent rounds. The Round 1 teacher sample consisted of a basic sample of new and 
experienced teachers and a supplemental sample of bilingual teachers selected independently of 
the basic sample. For the basic sample, the within-school teacher allocation involved the 
following steps: 

(1) Determine the total numbers of teachers to be selected from the lists provided by the sample 
schools. These numbers were chosen according to a formula that satisfied two requirements: 
(a) to make the product of the school's selection probability and the sampling fraction for 
teachers within the school constant within each school stratum; and (b) .to make the average 
number of teachers selected from schools in that stratum equal to a target number established for 
schools of that sector (public or private) and level (elementary, secondary, or combined). After 
this calculation, constraints were placed to sample at least one teacher per school and no more 
than twice the target average number of teachers for the sector and school level stratum. The 
target average number of teachers varied by sector and school level, ranging from an average of 
about three teachers for public elementary and private combined schools to an average of about 
seven teachers for public secondary schools. 

(2) Allocate the sample for each school between new and experienced teachers. In public 
schools, new and experienced teachers were allocated in proportion to the number of teachers of 
each type, as determined from the teacher listing forms. In private schools, new teachers were 
oversampled relative to experienced teachers by a fixed factor of 1 .6. The oversampling ensured 
a sufficiently large sample of new private school teachers in both the Teacher Survey and the 
subsequent Teacher Followup Survey. 

For each school, teachers were first sorted into new and experienced teachers and then by 
primary field of teaching as reported on the listing form. For elementary school teachers, the 
fields were general elementary, special education, or other teaching assignment. For secondary 
school teachers, they were mathematics, science, English, social studies, vocational education, or 
other teaching assignment. For combined schools, teachers were sorted by grade level and then 
by the corresponding fields for elementary or secondary teachers. Following the sorting 
operation, a systematic sample of teachers was selected with equal probability within the groups 
of new and experienced teachers at the rate determined for that group in the school. 

The Round 1 supplemental bilingual teacher sample was selected independently from the basic 
sample. The supplemental sample was designed so that the sample size for bilingual teachers 
from the basic sample and supplementary sample combined would be sufficient to provide 
separate estimates for California, Texas, Florida, Illinois, New York, and all other states as a 
group. Within each school, teachers were sorted into new and experienced teacher groups and 
then by primary field of teaching. The supplemental sample was selected by systematic 
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sampling. To avoid overburdening schools, if the sum of the basic and bilingual teacher samples 
was greater than 20 in any school, the bilingual sample was systematically reduced to make the 
total teacher sample equal to 20. 

Round 2. In Round 2, teachers on the list from each sample school were sorted into one of five 
teacher groups: API, AIAE, bilingual, new, and experienced. Teachers falling in more than one 
of these categories were placed in the first group listed. For example, an Asian bilingual teacher 
would be allocated to the API stratum. Teachers within each school and teacher group stratum 
were sort by primary field of teaching as in Round 1 and a systematic sample of teachers was 
selected with equal probability at the rate determined for that group. In private schools, new 
teachers were oversampled relative to experienced teachers by a factor of 1.8. As in Round 1, 
the number of new and experienced teachers was limited to be no more than twice the target 
number of teachers for the sector and school level stratum. 

Determine the number of bilingual, API, and AIAE teachers to be selected from each school. 
The numbers of bilingual, API, and AIAE teachers to sample from the list provided by each 
sample school were determined using a formula similar to that used for new and experienced 
teachers. The first requirement was to make the product of the school's selection probability and 
the sampling fraction for teachers within the school constant within each school stratum. The 
second requirement was to meet the target sample size desired for the group. The difficulty in 
estimating the national sampling intervals for these groups of teachers and the sample reduction 
approach used to handle this problem were outlined in section 5.3. When the combined sample 
of new, experienced, API, AIAE, and bilingual teachers exceeded 20 in a school, the API, AIAE, 
and bilingual teachers were proportionally reduced to meet the maximum requirement. 

Round 3. In Rounds 1 and 2, the teacher samples were selected separately for each sample 
school. To give greater control on the sampling process — particularly for API, AIAE, and 
bilingual teachers — Round 3 teachers were sampled from combined lists of teachers for sets of 
schools. Sampling was conducted in three waves: in mid-December, mid-January, and mid- 
February. The sampling by wave prevented the straggling teacher listing forms from delaying 
the whole teacher sampling process (Abramson et al., 1996). At the end of the first wave, due to 
the higher than expected listing form response rate, the projected total sample size was found to 
be higher than expected. To compensate, the sampling rate was adjusted so that the expected 
average number of teachers per school was reduced for the subsequent waves of teacher 
sampling. 

The teacher list frames for each of the three waves were sorted by school strata and, within each 
stratum, by school order of selection. Within schools, teachers were sorted by the five teacher 
strata and by primary fields of teaching, as in Round 2. Within each school and teacher stratum, 
teachers were selected systematically throughout the stratum across schools to produce an overall 
equal probability sample of teachers for that stratum. 

Sample size. Table 5-1 shows the numbers of teachers in the samples for Rounds 1 to 3. The 
sample in each round is in excess of 65,000 teachers. The numbers of new and experienced 
teachers for Round 1 include bilingual, API, and AIAE categories who were counted in separate 
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Table 5-1. — Sampled teachers for the Teacher Survey, by type: Rounds 1-3 



Sample sizes 


Teacher groups 


Round 1* 
(1987-88) 


Round 2 
(1990-91) 


Round 3 
(1993-94) 


Total 


67,771 


65,217 


68,284 


Experienced 


58,275 


52,084 


55,789 


New: Public schools 


6,633 


5,970 


4,799 


Private schools 


2,863 


2,002 


2,182 


Bilingual 


3,129 


2,121 


2,118 


American Indian, Aleut or Eskimo (AIAE) 




1,529 


1,661 


Asian or Pacific Islander (API) 




1,511 


1,735 



'The numbers of new and experienced teachers in Round 1 included bilingual, API, and AIAE teachers. 
*Not applicable. (Not listed as a separate category.) 

SOURCES: NCES (1991d); Gruber, Rohr, and Fondelier (1993, 1996). 



strata in subsequent rounds. Of the 3,129 bilingual teachers in the Round 1 sample, 682 teachers 
were selected in the basic sample and 2,447 teachers were added through the bilingual 
supplement. The sample sizes for new and experienced teachers in Rounds 2 and 3 do not 
include bilingual, AIAE, and API teachers. 

Design effects. As explained in section 2.4, the complex sampling design of SASS produces 
sampling variances different from those produced by a simple random sample of the same 
sample size. The impact of the complex design on the reliability of a sample estimate, relative to 
a simple random sample, can be measured by the design effect (deff). This measure reflects 
increases in the variances arising from the clustering and departures from an equal probability 
sample and decreases from the use of stratification and the estimation procedures. A deff larger 
than 1 indicates that the sampling error is higher than that from a simple random sample of the 
same size. 

Salvucci, Weng, and Kaufman (1995) computed deffs for groups of statistics by characteristics 
for the Round 2 Teacher Survey. Table 5-2 shows the deffs by sector for groups of statistics on 
teacher totals (e.g., number of male teachers), teacher averages (e.g., average number of years as 
a part-time teacher), and teacher proportions (e.g., proportion of married teachers). The average 
deff for totals is high because one item (number of white teachers) had a very high deff. 

5.4.3 Frame coverage and evaluation studies 

The coverage of the Teacher Survey depends on the coverage of the sample frame for schools 
(Quality Education Data [QED] or Common Core of Data [CCD]), which is discussed in section 
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Table 5-2. — Average design effects for estimates of totals, averages, and proportions, for 
selected items in the Teacher Survey: Round 2 



Average design effect (deff)* 



School sector 


Total 


Average 


Proportion 


Public schools 


4.95 


3.45 


2.84 


Private schools 


3.32 


2.33 


1.90 


Number of estimates 


16 


20 


23 


♦Design effects were not available for Rounds 1 and 3. 
SOURCE: Salvucci, Weng, and Kaufman (1995). 



2.4.3. It then depends on the completeness of the teacher listings provided by sample schools. 
This section discusses issues of list coverage and evaluation studies to improve the teacher listing 
forms. 

Coverage. Since the teacher listings were prepared early in the school year, only those teachers 
who were on the school’s rolls at that time would have been listed and eligible for sampling. 
Teachers who joined during the school year are not reflected in the Teacher Survey. 

Another issue is that a school's roll of teachers might have included some teachers who were on 
leave during the initial part of the school year. The instructions for the teacher listing forms did 
not specify whether or not teachers on leave should be included. The extent of this problem is 
not known. 

Some teachers recorded on a school's listing form had left the school by the time of the Teacher 
Survey data collection. No attempts were made to obtain completed questionnaires from 
sampled teachers of this type; they were classified as out of scope. 

Teacher list nonresponse. Some sample schools that were eligible for SASS did not provide 
teacher lists. As a result no teachers from these schools were included in the sample. These 
teachers are reflected through an adjustment in the teacher weights. For private schools, the list 
nonresponse rate was 12 percent in Round 1, 10 percent in Round 2, and 9 percent in Round 3. 
For public schools, the list nonresponse rate ranged between 4 and 5 percent in each round. 

Out of scope. Some teachers sampled from the teacher lists were later found to be ineligible for 
the Teacher Survey (see section 5.2). In Round 2, the out-of-scope rates were 7 percent of the 
sample teachers in public schools and 12 percent of the sample teachers in private schools 
(Gruber, 1992b). In Round 3 the correspon^ng rates were 5 percent for public schools and 10 
percent for private schools (Monaco, Salvucci, Zhang, Hu, and Gruber, 1998). Section 5.2 
discussed the stages in which an ineligible person can be excluded from the teacher survey. 

Discrepant teacher count reported by the school in the School Survey. The count of eligible 
teachers reported on the school questionnaire was not always the same as the number of teachers 
recorded on the teacher listing form for that school. In Round 1, in an average state, there were 5 
percent fewer teachers recorded on the teacher listing forms than were reported on the school 
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questionnaires (Kaufman, 1991, p.67). For Rounds 2 and 3 the teacher weights were adjusted to 
make the estimates of teacher counts from the Teacher Survey consistent with the weighted 
teacher counts from the School Survey (see section 5.7.3). The values of the adjustment factors 
by weighting cell indicate that, on average, fewer teachers were recorded on the listing forms 
than were reported on the school questionnaires. 

Frame evaluation studies. The 1993 Teacher Listing Validation Study (Royce and Schreiner, 
1994, and Royce, 1995) was undertaken in school year 1992-93 to answer the following three 
questions: What kinds of problems do schools have in completing the teacher listing forms? For 
public schools, would the local education agencies (LEAs) be able to provide teacher listings that 
are more accurate than those prepared by the schools? What is the relative accuracy of teacher 
counts from the school questionnaires and the teacher listing forms? 

The first two questions were investigated with a sample of 300 piiyate schools, 290 public 
schools, and 254 LEAs (some LEAs had more than one sample school). The third question was 
addressed through a separate sample of 300 public schools and 290 private schools. All of the 
schools in these two samples were asked to complete teacher listing forms, and the LEAs in the 
first sample were asked to complete teacher listing forms for the sample school(s) in their 
districts. Various techniques involving personal and telephone reinterviews and reconciliation 
for a subsample of schools were used to investigate the study questions. 

The main conclusion was that both public schools and their corresponding LEAs made errors 
listing teachers. The common errors included omission of part-time and specialized subject- 
matter teachers, and over-inclusion of some nonteachers (e.g., guidance counselors, speech 
therapists, and librarians). In general public school listings were found to be more accurate than 
those of their corresponding LEAs. SASS has therefore continued to ask for teacher lists from 
the schools. Comparisons of teacher counts from the teacher listing records with those on the 
school questionnaires suggest that the teacher listing records gave more accurate counts for both 
public and private schools. Common errors in the teacher counts on the school questionnaires 
included omissions of part-time teachers and erroneous inclusion of librarians and pre- 
kindergarten teachers. 

As a result of the Teacher Validation Study, the teacher listing form for Round 3 was revised to 
provide more specific instructions regarding the teachers to include and to exclude. The new 
form also included a toll-free 800 telephone number, where schools could obtain assistance in 
completing the form. 

In preparation for Round 4, two cognitive studies sought further improvements to the teacher 
listing form used in Round 3. Jenkins and Von Thum (1996) interviewed the principals (or 
persons who would ordinarily be responsible for completing the form) from 19 schools using 
cognitive study techniques. They found that some instructions on the Round 3 form were not 
well defined. Based on their results, Zukerberg and Lee (1997) redesigned the teacher listing 
form and conducted a cognitive study with another 20 schools to evaluate new features proposed 
for the form. The proposed changes include format changes to ease readability, the use of color 
to enhance the overall appearance, and better navigation structure. Many of their formatting and 
style recommendations were adopted after a pretest for Round 4. 
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5.5 CONTENT OF THE TEACHER QUESTIONNAIRES 

The teacher questionnaires covered similar topics in Rounds 1 through 3 but with some minor 
changes in specific items for each round (see NCES, 1991c; Gruber, Rohr, and Fondelier, 1993, 
1996). Each questionnaire included nine sections: 

■ Current Teaching Status collected data on whether the respondent was a full-time or part- 
time teacher, his/her other duties at the school, and jobs or activities outside of school. 

■ Teacher Experience asked for the year of first teaching experience, the number of breaks in 
his/her teaching career, and the year in which he/she began teaching in the sample school. 

■ Teacher Training collected data on college degrees, teaching assignment fields, certification, 
the fields that the respondent felt best qualified to teach, college courses in mathematics and 
science, participation in in-service training, membership in a teachers union, and 
participation in programs for beginning teachers. 

■ Current Teaching Load obtained information on grade levels taught, types and numbers of 
classes, numbers of students, subjects taught, and number of hours per week spent on job and 
job-related activities. 

■ Perceptions and Attitudes Toward Teaching collected data on the respondent's attitudes and 
perceptions about his/her current teaching job and toward teaching in general. 

■ Future Plans inquired about retirement eligibility, how long the teacher planned to remain in 
teaching, and his/her career plan for the following school year. 

■ Incentives and Compensation obtained data on teaching salary, benefits, and other earned 
income. 

■ Background Information collected data on gender, race/ethnicity, age, marital status, number 
of dependents, and family income. 

■ Limited-English-Proficient Students asked about limited-English-proficient students in the 
teacher's classes. 

The contents of the teacher questionnaire can be used to provide measures of teacher quality and 
teacher qualifications. In a review of various methods for measuring teacher quality and teacher 
qualifications, Ingersoll (1996) concluded that data from the Teacher Survey can be used to 
determine several aspects of teacher quality, including teachers preparation (e.g. ability, 
education, training, and background), induction (entry, apprenticeship, and mentoring programs 
for new teachers), utilization (assignment to service chores and out-of-field teaching), 
compensation (salary), resources (class size, equipment, preparation time, aides), decision- 
making participation, and personal and professional development opportunities. 

A study of methods to measure teacher qualifications by Fabiano.(1999) concluded that teachers' 
self-report data in national surveys such as the Teacher Survey provide the most common and 
cost-effective way of collecting nationally representative data on teachers and their 



qualifications. For validation, the report proposed for consideration that a transcript study be 
included in the Teacher Followup Survey; this proposal has not been adopted in Round 4. 

Forms. In Rounds 1 and 2, there were two separate versions of the teacher questionnaire, one for 
public school teachers and the other for private school teachers. A third form was added in 
Round 3 for teachers in BIA schools, and a fourth form was added in Round 4 for teachers in 
charter schools. The contents of the different forms for each round were nearly identical. In 
Rounds 2 and 3, different lists of the organizations endorsing the surveys were presented on the 
front page of the questionnaires. One item, asking whether the teacher was worldng at the school 
on a contributed service basis (less than full salary or no salary), was used only for private school 
teachers' questionnaires. In Round 4, charter school teachers who worked in the school prior to 
its becoming a charter school were asked if they supported the conversion (NCES, 1999). 

Content changes between Rounds 1 and 2. Round 2 reduced the number of perception and 
attitude items used in Round 1 and added more items on professional activities. An initial item 
in the perception and attitude section asked teachers about their agreement or disagreement with 
each of a series of statements relating to their teaching environment; the number of such 
statements was reduced from 23 in Round 1 to 14 in Round 2. Conversely, for an item asking 
teachers to evaluate the relative seriousness of various kinds of problems in their school, the 
number of categories was increased from 13 to 22. A new item asking teachers to rank the 
relative importance of various educational goals was added in Round 2. 

In the teacher training section, items on membership in professional organizations and 
participation in teacher induction programs (assistance to new teachers by mentor or master 
teachers) were added in Round 2. In the incentives and compensation section, a Round 1 item on 
incentives asked, with respect to each of several possible kinds of pay incentives, whether 
teachers favored them and whether they were receiving them. In Round 2, teachers were asked 
only to report which ones they were receiving. 

Content changes between Rounds 2 and 3. The Round 3 teacher questionnaires added and also 
deleted several items, based on field test and cognitive studies that evaluated response rates, 
ambiguities in the item wordings, burden on the respondent, and whether the data were needed 
for education research. New items included advanced (master's) degree, whether teacher was a 
Chapter 1 teacher, participation in in-service training, participation in a professional 
development program, teaching certificates awarded by other states or for fields other than the 
teacher's primary and secondary teaching assignments, number of tardy students, whether the 
teacher had been physically attacked by a student, and the year that the teacher would be eligible 
to retire. Items deleted include levels (elementary, secondary) at which the teacher had ever 
taught, numbers of college courses taken in teaching methods and in subjects currently taught, 
whether the teacher received pay incentives, main activity in the previous school year, and other 
items about students. 

Round 4 teacher questionnaires. The 1999-00 teacher questionnaire expands data collection on 
teacher preparation, induction, organization of classes, and professional development. It also 
collects data on use of computers. 



5.6 DATA COLLECTION 



The Teacher Survey, like the other SASS components, is conducted by mailed questionnaire 
with telephone followup of nonrespondents. This section discusses the data collection 
procedures (5.6.1); measurement errors examined in cognitive studies in preparation for each 
round of the survey (5.6,2); a teacher transcript record check study (5.6.3); reinterview studies to 
measure response variance (5.6.4); and nonresponse (5.6.5). 

5.6.1 Data collection 

The Teacher Survey used slightly different data collection procedures at each round. For Rounds 
1 and 2, questionnaires were mmled to the sample teachers at their schools during January and 
February (of 1988 for Round 1 and 1991 for Round 2). In February and March, a second 
questionnaire was sent to each sample teacher who had not yet responded. In March through 
June, U.S. Census Bureau field representatives initiated telephone followups of nonrespondents, 
calling either from their homes or from the U.S. Census Bureau's regional offices. The field 
representatives were instructed to call teachers at the schools to attempt interviews during 
nonteaching hours, for example, 8:00 a.m. to 9:00 a.m. and 3:00 p.m. to 5:00 p.m. For teachers 
unable to be interviewed at those times, they were to ask whether it would be possible to conduct 
the interview during a planning or free period. 

Special procedures were used for schools whose sample teachers had been identified only by 
alphanumeric indicators (Tl, T2, etc.). For these schools, the initial and followup questionnaires 
were mailed to the principal or other contact person at the school for distribution to the sample 
teachers. When telephone followups were necessary, the census field representative was 
instructed to call the principal or other contact person and ask that the teacher(s) who had not 
responded be requested to call the field representative in order to complete the interview. 

Procedures for Round 1 only. In Round 1, a school coordinator was appointed for each school to 
assist with both the distribution of questionnaires and the followup of nonresponding teachers. 
About 10 days after the initial mailing of questionnaires to the sample teachers, a letter was sent 
to each coordinator, listing the sample teachers and asking the coordinator to remind them to 
complete and return their questionnaires. About 6 weeks after the first mailing, replacement 
questionnaires for the nonresponding teachers were sent to the coordinator in a package. The 
coordinator was also contacted by telephone and asked to distribute the questionnaires to the 
teachers and encourage them to complete and return their forms (NCES, 1992). 

The use of school coordinators in Round 1 was based on the findings from a test that had 
demonstrated that response rates were higher for schools with coordinators, but that payment 
versus nonpayment of coordinators had no measurable effect on the results (Schwanz, 1987; 
Kaufman, 1988). The procedure was dropped in Round 2 in order to protect, to the greatest 
degree possible, the identity of the sample teachers in each school and, hence, the confidentiality 
of the data in the survey. 

Because of the relatively large number of teachers who did not mail back their questionnaires in 
Round 1, telephone followups were conducted only for a sample of those who had not 
responded. Rounds 2 and 3 followed up all mail nonrespondents. 
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Round 3. In Round 3, the teacher questionnaires were mailed in three batches, after each of the 
three stages of sampling mentioned above. About 67 percent of the questionnaires were mailed 
in mid-December (wave 1), 26 percent in early January (wave 2), and 7 percent in mid-February 
(wave 3). One week after each mailing, a reminder postcard was sent to all sample teachers. 
About 3 weeks after the postcard, a second questionnaire was sent to teachers who had not 
responded. It was necessary to send second questionnaires to about two-thirds of the public 
school teachers, as compared with three-quarters of private school teachers, and one-half of the 
BIA school teachers. 

Interviewers in the U.S. Census Bureau's telephone interviewing centers attempted computer- 
assisted telephone interviews (CATI) with private, BIA, and designated public school teachers 
who failed to respond by mail. Field representatives in the U.S. Census Bureau's regional offices 
attempted to telephone the remaining public school teachers and teachers for whom CATI 
interviewers were unable to collect the data (e.g., when the provided telephone number was 
incorrect). The field representatives' interviews used paper copies of the questionnaire as in 
previous rounds. The designation of public school teachers for CATI and non-CATI followup 
was done at the sampling stage as part of an experiment to evaluate data collection and mode 
effects (see following discussion on Mode effect). 

Table 5-3 shows the numbers and percent of sample teachers in each sector requiring telephone 
interview followup in Round 3. By design, about one-third of the public school nonrespondents 
to the second questionnaire were followed up by local field representatives. In the other sectors, 
the corresponding proportion was about one in eight. 



Table 5-3. — ^Teachers requiring and completing telephone interview followup, by followup 
interview and type of school: Round 3 



Sector 




CATI followup 






Non-CATI followup 




Total 




Completed 




Total 


Completed 




Number Percent' 


Number Percent^ 


Number 


Percent^ 


Number Percent'* 


Public 


13,241 


24 


7,001 


53 


6,690 


12 


4,169 


62 


Private 


4,629 


40 


2,081 


45 


698 


6 


161 


23 


BIA 


253 


36 


151 


60 


31 


4 


17 


55 



‘Percent of sample cases in CATI followup. 

^Percent of sample cases in CATI followup that were completed. Cases identified as noninterviews and out of scope for the survey during the 
CATI followup are not included. 

^Percent of sample cases in telephone followup by field representatives. 

^Percent of sample cases in telephone followup by field representatives that were completed. Cases identified as noninterviews and out of scope 
for the survey during followup are not included. 

NOTE: Numbers do not total because of noninterview out-of-scope cases and because nonresponse cases may be switched between CATI and 
non-CATI followup. 

SOURCE: Gruber et al. (1996). 



Mode effect. Round 3 included an experimental study to evaluate the effects of CATI on the 
quality of data obtained from public school teachers (Cole, Abramson, Parmer, and Schwanz, 
1997). This experiment, instituted at the sample selection stage, randomly designated about two- 
thirds of the public school sample teachers for centralized CATI followup (the CATI treatment 
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group), and the remaining one-third for telephone followup interviewing by field representatives 
in U.S. Census Bureau regional offices (the non-CATI treatment group). This assignment was 
done independent of , the actual response mode. The final disposition showed that about 69 
percent of the public school teacher interviews were returned by mail and, hence, needed no 
telephone followup, 19 percent were completed by CATI, and .12 percent by field representative 
telephone interviews. 

The study compared weighted estimates for various items in the questionnaire between the two 
treatment groups, CATI and non-CATI, for three subsets of respondents: all respondents, mail- 
return respondents, and telephone followup respondents (CATI or field representative). The 
main comparisons of interest were among the telephone interview respondents, between those 
designated and interviewed by CATI, and those designated non-CATI and interviewed by 
telephone interviewers using the paper-and-pencil questionnaire. The controls (or placebo 
groups) were the mail-return respondents and the comparison between the designated CATI and 
non-CATI cases among them. 

Significance tests were conducted to compare the responses obtained by the two designated 
treatment groups using a 10 percent significance level. The results (see table 5-4) showed 
significant differences for 18 percent of the items for all respondents, 14 percent for mail 
respondents only, and 29 percent for telephone respondents only as compared with the 10 
percent expected by chance. The percent of significant differences for mail respondents is, not 
surprisingly, not much greater than expected by chance. The higher percent of significant 
differences for the other two sets of comparisons suggests mode effects. Many of the items for 
which these estimates from the two treatment groups were significantly different pertained to the 
teachers' perceptions and attitudes about their schools. Overall, the CATI respondents provided 
a more positive response than the respondents interviewed by the field representatives. For 
example, more of the CATI respondents reported that their principals let them know what was 
expected from them and their schools' administrations treated them fairly and were supportive. 



Table 5-4. — Percent of items where weighted estimates from the CATI/Non-CATI treatment 
groups were significantly different: Round 3 







Percent of items 




Treatment group 
comparison 


All respondents 
(100 percent) 


Mail-return 
respondents 
(69 percent) 


Telephone interview 
respondents 
(31 percent) 


CATI/non-CATI 
weighted estimates 








with significant 
difference* 


18 


14 


29 



♦Significant difference at 10 percent significance level. 
SOURCE: Cole, Abramson, Parmer, and Schwanz (1997). 



Supervision of field representatives and quality assurance. As in most of the other SASS 
surveys, one method of controlling the quality of the data collection operations was through 
regional office reviews of the questionnaires completed by interview with mail nonrespondents. 
In the regional office reviews, the first four Teacher Survey questionnaires completed by each 
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field representative were clerically checked. If the total number of errors (including omissions) 
for the four questionnaires was 10 or more, additional questionnaires were reviewed until there 
were 4 consecutive questionnaires with a total of fewer than 10 errors. For the other 3 basic 
SASS surveys, the standard used for this review was fewer than "10 total errors in 2 
questionnaires (Gruber, Rohr and Fondelier, 1993, chapter VII). 

U.S. Census Bureau field staff comments on the collection procedures. Following completion of 
field work for Round 2 of SASS and again after the completion of the 1992 pretest for Round 3, 
each census regional office was asked to submit its comments and suggestions on how the survey 
instruments and collection procedures could be improved. Among the most frequent suggestions 
relating to the Teacher Survey were the following (U.S. Census Bureau, 1991a, 1992): 

B Distribute the questionnaires earlier in the school year, so that teachers and school 
administrators would not be asked to complete them during the period leading up to 
graduation, which is one of their busiest times of year; 

B Develop a procedure for conducting telephone followup interviews with teachers by calling 
them at their homes. Completing telephone interviews during school hours proved difficult 
because of the length of the questionnaire, the sensitivity of some of the items, and limited 
access to telephones in many schools; and 

a Make the cover page of the Teacher Survey questionnaire "friendlier" by including more 
information specifically addressed to the teacher, such as the letter from the Commissioner of 
NCES. 

A procedure for conducting telephone followup interviews with teachers at their homes was 
tested in the pretest for Round 3, as described in section 5.6.2. 

Length of time to complete questionnaire. The teacher questionnaires for Rounds 2 and 3 include 
a final question, "Not counting interruptions, how long did it take to complete this survey?" For 
questionnaires completed and returned by mail, the responding teacher completed this item; for 
questionnaires completed in followup telephone interviews, the interviewer completed the item. 
For Round 2, the median time for completion was 45 minutes for public school teachers and 40 
minutes for private school teachers, with interquartile ranges of 30 minutes for both groups. 
About 99 percent of all teachers completed the questionnaire in less than 2 hours. For Round 3, 
the same questions were asked, however, data were unavailable for analyses. 

5.6.2 Cognitive studies and pretesting 

Before each round of the SASS survey, NCES conducts field tests, cognitive studies, and 
reinterviews as part of the continuing effort to improve survey quality. Section 2.6.2 summarizes 
the activities conducted for the School Survey and other SASS components. This section 
describes these activities for the Teacher Survey. 

Round 1 pretest reinterviews. As described in section 2.6.2, the first reinterviews for the Teacher 
Survey occurred in conjunction with a large-scale pretest for Round 1 of SASS in the early part 
of 1987. The pretest, which covered 10 states, included 2,300 teachers in 220 public schools and 
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600 teachers in 75 private schools. The questionnaires were distributed by mail. A systematic 
sample of 127 teachers who responded to the mail questionnaire was selected for telephone 
reinterview by U.S. Census Bureau field representatives. Reinterviews were successfully 
conducted for 121 of them, for a 95 percent completion rate. In the reinterviews the teachers 
were asked how they had interpreted and answered selected questionnaire items, the extent to 
which they used or could have used records in responding, how much confidence they had in the 
accuracy of their responses, and whether they had any recommendations for improving the 
questionnaire. 

The report of the reinterviews (Nash, n.d.) included several recommendations for improvements 
in specific questionnaire items: 

■ In answering questions about college-level courses taken, about one-half of the teachers 
reinterviewed said they had referred to records and a large proportion of the rest said they 
had records available, As a result of this finding, an instruction was added to the relevant 
items on the final questionnaire for Round 1, "Please refer to records if you cannot accurately 
recall your coursework." (For further information on the accuracy of responses to questions 
about degrees and courses taken, see the Teacher Transcript Study, section 5.6.3). 

IB The pretest question on mathematics and science courses asked for number of credit hours 
completed. Most teachers felt it would be easier to report number of courses, and this 
recommendation was followed on the final questionnaire for Round 1. 

■ For a question on how the teacher's classes were organized, comments in the reinterviews led 
to the addition of a new category, pull-out classes (where teachers provide instruction to 
students who are released from their regular classes), to the final version of this item for 
Round 1. 

a For an item on time spent in school-related activities, the reinterviews showed that about two 
in five teachers, in responding, had not included time spent away from school on such 
activities. As a result, separate sub-items covering time for certain kinds of activities away 
from school were added to the final version. 

1990 pre-Round 2 cognitive interviews. U.S. Census Bureau staff members, using an early 
version of the 1990 pretest questionnaire, conducted "think-aloud" interviews with 20 teachers, 
10 from public schools and 10 from private schools, representing various grade levels and 
specialties (Bates and DeMaio, 1990). The teachers were asked to verbalize their thoughts as 
they completed the self-administered questionnaires. The census staff members asked questions 
as needed to understand how the teachers were interpreting the questions and what they were 
including in their answers. Findings from these interviews and from the subsequent pretest 
significantly influenced the development of the final questionnaires for the Round 2 Teacher 
Survey. 

The U.S. Census Bureau staff members observed that teachers often failed to follow skip 
instructions (telling them to pass over questions that did not apply to them) and that they 
frequently failed to check boxes for "none," either leaving the item blank or entering "0" in the 
space reserved for an amount or number. For the most part, these errors were unlikely to bias the 
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survey estimates, because the correct responses could be inferred, during data processing, from 
other entries on the questionnmre. However, the attempts by the teachers to answer questions 
that did not apply to them caused frustration that could lead to a decline in the perseverance 
needed to provide complete and accurate answers. 

One possible solution to the problem of the failure to skip inapplicable questions is to include 
redundant instructions, for example, using both a skip instruction next to the answer spaces in the 
item where the skip begins (the branching item) and an instruction at the beginning of each item 
that applies only to respondents with certain characteristics. The pretest questionnaire had some 
redundancy of this kind; the U.S. Census Bureau staff interviewers recommended additional use 
of redundant instructions. 

Numerous problems were observed with an item for teachers who were teaching subject-matter 
(departmentalized) courses to different groups of students. This item asked responding teachers 
to enter, in a matrix format, several items of information for each separate class they had taught 
in the most recent full week of teaching. The instructions for the items to be reported for each 
class appeared on the page preceding the matrix for recording the items. 

The word "class" itself caused difficulty because some teachers interpreted it to mean class 
period rather than, as was intended, a group of students receiving instruction in a subject during 
one or more class periods in the reference week. One of the specific items requested for each 
class was the number of graduation units associated with it. The instructions for this item asked 
the teacher to enter a code (0 for no credit, 1 for less than one unit, 2 for one unit, etc.), but some 
teachers entered the actual number of units rather than the code, an error not likely to have been 
detected in data processing. 

Difficulties were also observed for an item about hours spent on school -related activities during 
and after school hours. As noted above, a similar item in Round 1 had numerous response 
problems and the responses were not included in the public-use data tapes. The main kinds of 
problems noted were the following: 

Q Answers were given in terms of hours per day, rather than for a reference week; and 

0 Errors due to misunderstanding of the question format, which called for reporting time spent 
during and after school hours separately and, in each case, providing a total and a breakdown 
into two or more categories. 

This content of this item was substantially reduced and modified in the final questionnaire for 
Round 2. 

The U.S. Census Bureau staff reporting on these interviews made several recommendations for 
changes in specific items and for additional research on some of the kinds of response problems 
that were observed. Many of their recommendations were adopted for implementation in 
Round 2. 

Item-by-item review in 1990 (pre-Round 2). Over 900 Teacher Survey questionnaires from the 
1990 SASS field test (pre-Round 2) were reviewed to identify items that were misunderstood by 



respondents or were difficult for them to answer (Fondelier and Bynum, 1990). The reviewers 
found several indications that respondents were concerned about the length of the questionnaire; 
notes to this effect on the forms, partially completed questionnaires and information on reasons 
for refusals. They also observed that the quality of data for mail responses appeared to be much 
better than that of the responses obtained by telephone followup. This was attributed in part to 
the unsuitability of the questionnaire design and format for telephone interviews, especially 
when the interviews had to be completed with teachers at their places of work, and in part to 
failure of the telephone interviewers to follow skip instructions and to complete items correctly 
and legibly. 

The specific item found by the reviewers to have the most problems asked for information about 
classes taught in departmentalized courses (see also the comments on this item in the preceding 
discussion of cognitive interviews). Several respondents misinterpreted the meaning of "class" 
and several clearly failed to read the instructions that appeared on the page preceding the one on 
which the answers were to be recorded. For some elementary music, art, and physical education 
teachers, insufficient lines were provided to record the data requested for each of their classes. 

The reviewers made numerous specific proposals for changes in the wording and format of 
questions and instructions. Many of their recommendations, as well as some of those emanating 
from the cognitive interviews, were followed in the final questionnaires for Round 2. 

1991-92 Round 3 pretest. The 1991-92 field test teacher sample consisted of 420 public school 
teachers and 480 private school teachers; the response rate of teachers from both school sectors 
was over 96 percent. The teacher field test was used to evaluate changes to the teacher 
questionnaire: items to delete, to revise, and to add to the Round 3 teacher questionnaires. Since 
prior experience had shown that U.S. Census Bureau interviewers had difficulty in conducting 
telephone interviews with mail nonrespondents at school, a Round 3 pretest included a test of an 
alternative approach. In that pretest, which was conducted early in 1992, a postcard was sent to 
each sample teacher who had not responded within about 2 weeks of the second mailing. The 
message included the following: 

If we do not receive your completed questionnaire by mail within 2 weeks, we 
will contact you by telephone at your school to collect this information. If you 
prefer to be contacted at home, please provide your home telephone number on 
the attached postcard and return it in the next few days. 

Teachers who supplied their home telephone numbers were contacted for interviews at their 
homes (Ferrell, 1992). Only a small proportion of teachers returned the postcards (only 17 of the 
350 teachers to whom they were sent). This procedure was therefore not adopted for followup of 
mail nonrespondents in Round 3. 

Pre-Round 4 cognitive studies. A large pretest was conducted in the fall of 1998 for Round 4^ 
the results of which will appear in the Data File User's Manual for Round 4. As part of the 
redesign of the questionnaires for Round 4, cognitive interviews were conducted with 20 
teachers (Jenkins, 1997). The goals of this study were to determine the effectiveness of new 
formats and how well respondents comprehended the questions. The interviews were conducted 
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at the U.S. Census Bureau's cognitive laboratory with 15 teachers from different school types 
and with various teaching experience, and in New Jersey with 5 teachers who were identified by 
the State Department of Education as having alternative certificates. The tatter 5 cognitive 
interviews were conducted because the answers of such teachers to a question about the type of 
certificate were considered to be problematic. 

Various cognitive techniques were used, including the concurrent think-alOud technique, the use 
of paraphrasing, and unstructured retrospective interviewing that excluded sections that did not 
contain problematic items. Respondents were asked to complete a shortened teacher 
questionnaire. They were asked to read aloud as they read through the questionnaire and to think 
aloud as they answered the questions. The interviews took between 30 and 90 minutes to 
complete. They were tape-recorded with the respondent’s permission, and a transcript or a 
summary was prepared for each interview. 

An item-by-item review of the interview results identified some formatting issues and problems 
with skip instructions. The proposed changes to the questionnaire included the use of color 
backgrounds (blue for the private school questionnaire and green for the public school 
questionnaire) and white spaces to offset answer space, the use of a one-column vertical format, 
and skip instructions to help navigate the respondents through the questionnaire. 

5.6.3 A teacher transcript record check study 

In Round 2 of SASS, an experiment was undertaken to compare the accuracy of teachers' self- 
reports about their educational backgrounds with data obtained from transcripts of their college 
records (Chaney, 1993, 1994). The data items compared for the two methods included degrees 
awarded, year of award, major and minor fields of study, and number of courses taken or credit 
hours earned in four separate areas: education, area of main teaching assignment, area of second 
teaching assignment, and science and mathematics. 

The study was carried out "off-line," that is, a separate sample of teachers was used for the 
experiment. Two versions of the Teacher Survey questionnaire were administered, one asking 
for information on the number of courses taken and one asking for information on the number of 
credit hours earned in the relevant fields. Out of the initial sample of 867 teachers, 32 were later 
found to be ineligible for the study. Of the 835 eligible teachers, 592 (71 percent) agreed to 
participate in the study and provided names of the colleges they had attended, so that transcripts 
could be requested from these colleges. 

The teacher questionnaires were administered by the U.S. Census Bureau, using the standard 
mailing and telephone followup procedures. The request for permission to obtain respondents' 
transcripts came at the end of questionnaire. Thus for telephone interviews, the knowledge that 
this was to be done was unlikely to have influenced responses about degrees and courses. It is 
possible that some of the mail respondents could have gone back and checked their responses to 
these items after they discovered that their transcripts would be obtained. 

A total of 1,835 transcripts was requested. A transcript was provided for 74 percent of these 
cases and for 3 percent the college said it had no record of the identified person having attended. 
(For about one-third of the latter group, the teacher's attendance at the college could be 
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confirmed on the basis of transfer notations on another college's transcript.) In 4 percent of the 
cases the college said it could not locate the records and for the remaining 19 percent the college 
did not respond to the request. The colleges also provided 168 transcripts that were not 
requested. These were generally instances where the same person had both undergraduate and 
graduate work at the institution but did not report both on his or her questionnaire. 

For the 592 sample teachers who participated in the study, all requested transcripts were obtained 
for 51 percent, some but not all transcripts were obtained for 41 percent, and no transcript 
information was obtained for the remaining 8 percent. For some of the data items, such as 
degrees awarded, partial transcript information was sufficient to confirm self-reported data. 
However, if a teacher's self-reported degree was not confirmed, it would be difficult to conclude 
that the self-report was incorrect unless all requested transcripts for that teacher had been 
received. 

The data from the comparisons indicated that self-reports of types and years of degrees earned 
and major fields were, for the most part, accurate. However, information on numbers of courses 
and credit hours was less accurate. The study report says the following: 

Other errors appeared to show bias on the part of the respondent. For example, though 
there were errors in both directions, the general pattern was for teachers to overstate their 
preparation in their second teaching assignment and in mathematics and science as 
compared with the records on their transcripts. Since courses were coded as falling 
within the specified areas if there were any ambiguity, this overstatement is the reverse of 
what might be expected if there were simply differences between the teachers and coders 
in how to classify courses (Chaney, 1994, p.20). 

Any proposal to rely on transcripts as the primary source of information on courses for sample 
teachers would, of course, have to take into account the additional costs associated with the 
collection of transcript data and the likelihood of higher item nonresponse resulting from failure 
of teachers to report all of the colleges where they had taken courses and failure of some colleges 
to supply the requested transcripts. 

5.6.4 Reinterviews 

After each round of SASS, reinterviews have been attempted for about 1,100 teachers, or about 1 
in 60 of the total sample. They were successfully completed for about 75 percent of the eligible 
cases in Round 1, 83 percent in Round 2, and 73 percent in Round 3. All reinterviews of 
teachers in Rounds 1 and 2 were conducted by telephone. Round 3 used the same mode at 
reinterview as the original survey; about two-thirds of the cases were completed by mail and the 
remaining one-third were completed by telephone followup (either centralized CATI or field 
representative interviews). 

Topics covered in the reinterviews. There were many differences between rounds in the topics 
covered in the Teacher Survey reinterviews. The four topics covered in both Rounds 1 and 2 
were educational attainment, full- and part-time teaching experience by sector (public and 
private), current teaching assignment, and plans to continue teaching. However, there were some 
significant changes in question wording or format of items and few items used in Round 1 were 
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replicated in Round 2. The Round 3 reinterview covered six areas: teaching assignment and 
certifications, Chapter 1 status, teacher activities, in-service or professional development 
programs, student behavior problems, and teacher salary and benefits. 

Table 5-5 presents reinterview findings for both Rounds 1 and 2 for the questions on teachers' 
educational attainment. The format for the questions on educational attainment was substantially 
revised between the two rounds. In Round 1, a single "mark all that apply" question was asked 
for the following: associate degree or vocational certificate, bachelor's degree, second bachelor's 
degree, master's degree, second master's degree, education specialist or professional diploma (at 
least one year beyond Master's level), doctorate (e.g., Ph.D., Ed.D.), first professional degree 
(e.g., M.D., L.L.B., J.D., D.D.S.), and no degree or diploma. Round 2 had two single "Yes/No" 
questions: Do you have a bachelor's degree?; do you have a master's degree?, and a "mark all 
that apply" question for degrees such as associate degree, education specialist/a professional 
diploma, and doctoral or first professional degree. 

Table 5-5. — Measures of inconsistency for educational qualifications in the Teachers Survey: 
Rounds 1 and 2 



Educational attainment 
items 


Percent mention 
(survey interview) 


Gross 

difference rate 


Index of inconsistency 


Round 1 
(1987-88) 


Round 2 
(1990-91) 


Round 1 
(1987-88) 


Round 2 
(1990-91) 


Round 1 
(1987-88) 


Round 2 
(1990-91) 


Bachelor's degree 


98 


98 


8* 


1* 


80 


— 


Master's degree 


42 


41 


4* 


1* 


g* 


2* 


Prof. dipl./Ed. spec. 


4 


5 


7 


5 


70 


63 


Associate degree 


14 


7 


8 


7 


37* 


54* 



”Not available. (Not computed because there were too few cases without a bachelor's degree.) 
^Statistically significant difference between Round 1 and Round 2 (at a 10 percent significance level). 
SOURCE: Bushery, Royce, and Kasprzyk (1992). 



The findings in table 5-5 are expressed in terms of the gross difference rate (the percent of 
discrepant answers for the item between the original response and the reinterview response) and 
the index of inconsistency (see section 2.6.4). As was the case for the School Principal Survey 
(see section 3.3 and table 3-8), the data suggest that the revisions led to more reliable and 
accurate reporting on bachelor's and master's degrees in Round 2, even though deficiencies in 
Round 1 for teachers had been less serious than for school principals. The same could not be 
said for the reporting of associate degrees and educational specialist or professional diplomas; in 
fact, the index of inconsistency for associate degrees was somewhat higher in Round 2. 

Table 5-6 shows Round 1 and Round 2 reinterview results for questions on years of teaching 
experience, full- and part-time, in the public and private sectors. For this analysis, responses 
were classified into four categories: less than 3 years, 3-9 years, 10-20 years, more than 20 
years. In both rounds, the two full-time questions had relatively low response variance, but this 
was not true for the questions on part-time teaching. Despite the use of a redesigned format for 
the part-time questions in Round 2 (using 4 response categories instead of asking the respondents 
to fill in the number of years), no significant improvements were noted and the gross difference 
rate for part-time teaching in private schools actually increased somewhat. 
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Table 5-6. — Measures of inconsistency on the topic "years of teaching" in the Teacher Survey: 
Rounds 1 and 2 



Years of teaching experience* 


Gross 

difference rate 


Index of 
inconsistency 


Round 1 
(1987-88) 


Round 2 
(1990-91) 


Round 1 
(1987-88) 


Round 2 
(1990-91) 


Full-time, Public 


8 


7 


11 


10 


Part-time, Public 


9 


7 


44 


43 


Full-time, Private 


5 


5 


12 


9 


Part-time, Private 


^3 


^8 


39 


38 



*For this analysis, responses were classified into four categories: less than 3 ye^, 3-9 years, 10-20 years, more than 20 years. 
^Statistically significant difference between Round 1 and Round 2 (at a 10 percent significance level). 

SOURCE: Bushery, Royce, and Kasprzyk (1992). 



It is difficult to compare the reinterview results for Rounds 1 and 2 for the questions on teaching 
assignment since they were substantially changed. For plans to remain in teaching, the same 
question was used in both rounds and the estimates of the gross difference rate and the index of 
inconsistency were both significantly higher for Round 2. The gross difference rate rose from 40 
percent to 47 percent and the index of inconsistency rose from 55 to 67. 

Table 5-7 shows the distribution of estimated indexes of inconsistency for all items used in the 
Teacher Survey reinterviews in each round. In Round 1, about two-thirds of the items included 
were opinion questions and, as shown in the table, nearly all of them had indexes in the high 
range (values of 50 and over). These items covered teachers' views about topics like problems in 
their schools, their influence on school and classroom policies and practices, and the extent to 
which school administrators and other teachers had been helpful to them. In Round 2, only three 
opinion items were covered in the Teacher Survey reinterviews because it was felt that 
reinterview results for factual questions would be of more value for identifying problem 
questions and guiding efforts to improve their wording and format. 

One of the factual items included in the reinterviews in Round 2 asked teachers to report the 
grade levels for their current classes. There were 16 possible response categories, with an 
instruction to mark each one that applied. For the purpose of estimating indexes of 
inconsistency, each of the 16 categories was treated as a separate item. All of the 13 categories 
for which estimates could be made had indexes in the low range, which was not surprising for 
such a relatively straightforward item. The data in table 5-7 for factual items in Round 2 are 
shown with and without this item (item 29). When it is excluded, the distributions for factual 
items in Rounds 1 and 2 are somewhat similar. 

Aside from the opinion items and the topics covered in reinterviews for both rounds, which we 
have already discussed, the topics with high indexes of inconsistency, as measured by 
reinterviews, were pay incentives in Round 1 and nonteaching income, courses and certification 
in Round 2. 
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Table 5-7. — Number of items by index of inconsistency in the Teacher Survey: Rounds 1-3 





Round 1 
(1987-88) 




Round 2^ 
(1990-91) 




Round 3 
(1993-94) 




Index of inconsistency’ 


Factual Opinion 


Factual, 

all 


Factual, 
excl. 
item 29 


Opinion 


Factual Opinion 


Total questions 


20 42 


53 


37 


3 


78 


14 


Low (0-19%) 


3 0 


21 


8 


0 


11 


0 


Moderate (20-50%) 


4 3 


14 


14 


2 


30 


6 


High (51-100%) 


5 39 


10 


10 


1 


22 


8 


Not available^ 


8 0 


8 


5 


0 


15 


0 



‘Each item either had closed multiple-response categories or was converted to the equivalent by assigning class intervals to open-end responses. 
For items with more than 2 response categories, the L-fold index of inconsistency was estimated. 

^Did not meet the minimum requirements to compute a reliable estimate of the index of inconsistency. 

^Questionnaire item 29 asked about grade levels for the responding teacher's current classes, with 16 possible response categories. Since the 
teacher was asked to mark each category that applied, item 29 had to be treated as 16 separate items for the purpose of estimating indexes of 
inconsistency. 

SOURCES: Newbrough (1989); Royce (1994); Bushery, Schreiner, and Sebron (1998). 



The Round 3 reinterview study included a different set of questions than Rounds 1 and 2 (see 
Bushery, Schreiner, and Sebron, 1998). Among the questions evaluated, 30 questions had high 
indexes of inconsistency (indexes greater than 50). Items with high levels of inconsistency 
included two items on teacher certification, half of the items evaluated on teacher activities, 
several of the items on in-service or professional development programs, and several items on 
student behavior. The reinterview results suggested that some perception and opinion questions 
with high levels of inconsistency might be appreciably improved by using fewer response 
categories. For example, relative to items using four response categories (serious, moderate, 
minor, or not a problem), items that used only two categories (problem and not a problem) 
showed much less inconsistency. 

5.6.5 Nonresponse 

Unit nonresponse. Unit nonresponse could occur in the Teacher Survey for two reasons: a 
sample school could fail to provide a list of teachers for use in selecting the teacher sample (or, 
alternatively, fail to select a sample of teachers itself) or an acceptable questionnaire might not 
be obtained for a sample teacher. We will refer to these two sources of nonresponse as school 
list nonresponse and teacher nonresponse and their complements as school list response and 
teacher response. 

The school list response rate is the percent of sampled schools eligible for the School Survey that 
provided the teacher lists or selected the teacher sample. The teacher response rate is the percent 
of sample teachers eligible for the Teacher Survey who responded to the survey. The product of 
the school list and teacher response rates is termed the overall teacher response rate; it provides 
an indicator of the overall response rate for the Teacher Survey. 



The teacher response rate is confined to sample schools in which teachers were sampled, and 
excludes school staff who were sampled but who did not meet the survey definition of a teacher 
or who were no longer working at the sample school at the time the teacher questionnaire was 
distributed. In Round 2, 7 percent of those sampled from teacher lists in public schools and 12 
percent of those sampled in private schools were found to be out of scope. In Round 3, the 
corresponding percents were 7 percent in public schools and 10 percent in private schools. 

Table 5-8 shows the school list, teacher, and overall teacher response rates for public and private 
schools in Rounds 1 to 3. The school list response rates are unweighted whereas the teacher 
response rates are weighted by the teacher basic weight to reflect the probability of selection. 
Public schools have both higher school list and higher teacher response rates than private schools 
in all three rounds. As a result, the overall response rates for public schools were, on average, 
some 12 percentage points higher than those for private schools. 

Table 5-8. — School list, teacher, and overall response rates for the Teacher Survey, by sector: 
Rounds 1-3 







Response rate (percent) 




Sector and component 


Round 1 
(1987-88) 


Round 2 
(1990-91) 


Round 3 
(1993-94) 


Public 


School list* 


96 


95 


96 


Teacher^ 


86 


90 


88 


Overall^ 


83 


86 


84 


Private 


School list* 


88 


90 


91 


Teacher^ 


79 


84 


80 


Overall^ 


70 


75 


73 



‘Percent of schools eligible for the SASS School Survey that provided teacher lists for use in sampling teachers, unweighted. 

^Percent of eligible sample teachers responding to the Teacher Survey, weighted by the basic teacher weight to reflect the probability of selection. 
^Product of school list and teacher response rates. 

SOURCES: NCES (I99ld); Gruber, Rohr, and FondeUer (1993, 1996). 



The teacher response rates in table 5-8 are higher in Round 2 than Rounds 1 and 3, which was 
unexpected. Based on the results of the Round 1 pretest experiment with coordinators, one 
might have predicted lower teacher response rates in Round 2, when no coordinators were used, 
but the reverse occurred. Round 3 used a three-wave approach intended to allow more time for 
data collection from teachers. However, this approach did not improve the teacher response rate. 

Public school teachers. Table 5-9 shows the weighted teacher response rates for public school 
teachers by state for Rounds 1 to 3. With exception of the District of Columbia, whose response 
rate stood at around 70 percent for all three rounds, no state had a response rate of less than 80 
percent in Rounds 2 and 3, and only four states (Hawaii, Maryland, New York, and Rhode 
Island) had response rates below 80 percent in Round 1. In all three rounds a majority of states 
recorded response rates of 90 percent or higher. 
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Table 5-9. — Weighted response rates for public school teachers in the Teacher Survey, by state: 
Rounds 1-3 



State 




Weighted response rate (percent)* 




Round 1 
(1987-88) 


Round 2 
(1990-91) 


Round 3 
(1993-94) 


Total 


86 


90 


88 


Alabama 


90 


91 


90 


Alaska 


90 


90 


86 


Arizona 


91 


95 


90 


Arkansas 


91 


94 


91 


California 


84 


88 


82 


Colorado 


89 


95 


88 


Connecticut 


80 


86 


88 


Delaware 


86 


96 


86 


District of Columbia 


69 


69 


71 


Florida 


87 


89 


91 


Georgia 


87 


93 


92 


Hawaii 


74 


88 


86 


Idaho 


93 


95 


93 


Illinois 


91 


96 


87 


Indiana 


92 


95 


91 


Iowa 


93 


96 


92 


Kansas 


91 


96 


91 


Kentucky 


86 


89 


90 


Louisiana 


81 


93 


91 


Maine 


92 


90 


90 


Maryland 


74 


90 


88' 


Massachusetts 


85 


84 


87 


Michigan 


86 


85 


89 


Minnesota 


90 


94 


93 


Mississippi 


88 


93 


91 


Missouri 


88 


91 


92 


Montana 


91 


95 


92 


Nebraska 


93 


93 


92 


Nevada 


91 


89 


84 


New Hampshire 


85 


93 


90 


New Jersey 


81 


86 


86 


New Mexico 


85 


90 


90 


New York 


75 


79 


80 


North Carolina 


89 


96 


90 


North Dakota 


93 


96 


93 


Ohio 


88 


88 


89 


Oklahoma 


90 


94 


87 


Oregon 


94 


91 


90 


Pennsylvania 


88 


93 


88 


Rhode Island 


75 


87 


85 


South Carolina 


89 


91 


91 


South Dakota 


95 


95 


89 


Tennessee 


85 


93 


89 


Texas 


87 


92 


90 


Utah 


90 


98 


92 


Vermont 


87 


96 


86 


Virginia 


87 


91 


90 


Washington 


90 


88 


88 


West Virginia 


88 


95 


92 


Wisconsin 


89 


95 


93 


Wyoming 


91 


97 


91 



♦Basic weight is used to reflect the probability of selection. Rates restricted to schools providing teacher lists. 
SOURCES: NCES (1991d); Gruber, Rohr, and Fondelier (1993, 1996). 
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Private school teachers. Table 5-10 shows teacher response rates for private school teachers by 
association group and school types for Rodnds 1 through 3. In Round 1, the lowest rate 
observed, 58 percent, was for teachers in schools included in the area sample. For the list 
samples, response rates ranged- from 61 percent to 87 percent. In Round 2, the teacher response 
rate for the area frame schools was 74 percent, and the response rates for the list samples ranged 
from 57 to 95 percent. In Round 3, the response rate for area frame teachers was 71 percent, and 
the rates for the list samples ranged from 56 to 91 percent. 

Analytic studies were conducted for Rounds 2 and 3 to examine variations in response rates for 
different categories of school. Some results are displayed in table 5-11. The study by Scheuren 
et al. (1996), which investigated variations in response rates for the first four characteristics in 
the table for Round 2, used a Bonferroni multiple comparison test with a 10 percent significance 
level for each characteristic. In the table S denotes a significant difference in response rates 
between at least one pair of categories for the characteristic; NS denotes no significant 
differences. Some significant category differences were found by region and school size for both 
public and private schools, by urbanicity for public schools, and by school level for private 
schools. 

Monaco, Salvucci, Zhang, Hu, and Gruber (1998) performed similar analyses for the 1993-94 
Round, including the four characteristics examined in the 1990-91 study and adding the 
characteristics displayed in the lower part of table 5-11 (some of which related only to public 
schools, others only to private schools). They used an overall test of differences across 
categories; the significance level reached is indicated by the P-value in the table. The results for 
Round 3 exhibits similar pattern of significance/nonsignificance for the first four characteristics 
as Round 2. For private schools, a significant difference was found by urbanicity in Round 3 
that was not observed in Round 2. 

The new characteristics added for the 1993-94 analyses for public school teachers show a 
variation in response rate by school type (regular or nonregular), minority enrollment, new 
teacher (yes or no), and race/ethnicity. For private school teachers, statistically significant 
variations are found by race/ethnicity, whether the school was sampled in Round 2 of SASS in 
1990-91, and by the response status of the schools in the 1991-92 PSS. 

Item nonresponse. Table 5-12 summarizes unweighted item response rates for public and private 
school teachers in Rounds 1 to 3. Because of changes in questionnaire content and wording, the 
rates for the rounds are not directly comparable. The percent of items with response rates of 90 
percent or more were around 90 percent for Rounds 1 and 3, but were appreciably lower for 
Round 2. The minimum item response rate was appreciably lower in Round 1 than in 
subsequent rounds. The items involved were deleted or revised for the later rounds. 

The percent of items with response rates below 75 percent were typically very small. In Round 
1, two questionnaire items for both public and private schools had response rates below 75 
percent. The first of these was an item asking for the second major or the minor field of study 
for each degree reported. Teachers responding to the questionnaire were asked to enter a code 
00 if they did not have a second major or a minor field. Apparently, many of them simply left 
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Table 5-10. — Weighted response rates for private school teachers in the Teacher Survey, by 
association group and school type: Rounds 1-3 





Weighted response rate (percent)* 




Round 1^ 


Round 2 


Round 3 


Association group 


(1987-88) 


(1990-91) 


(1993-94) 


Total (area and list frames) 




79 


84 


80 


List frame 




— 


84 


80 


Area frame 




58 


74 


71 


Schools on list frame by association group: 

Association of Military College and Schools of United States 
National Catholic Education Association, 




85 


88 


89 


Jesuit Secondary Education Association 




84 


88 


83 


Friends Council on Education 




85 


87 


84 


National Association of Episcopal Schools 




82 


84 


84 


National Society of Hebrew Day Schools 






60 


63 


Solomon Schechter Day Schools 




► 64 


84 


78 


Other Jewish 






57 


56 


Lutheran Church — Missouri Synod 






95 


91 


Evangelical Lutheran Church — Wisconsin 




*- 87 


92 


91 


Evangelical Lutheran Church in America 


1 


87 


88 


Other Lutheran 


J 


1 


89 


84 


General Conference of Seventh-Day Adventists 




81 


82 


89 


Christian Schools International 




87 


90 


77 


American Association of Christian Schools 




61 


70 


64 


National Association of Private Schools for Exceptional Children 




71 


76 


81 


American Montessori Society Schools, other Montessori Schools 




79 


77 


72 


National Association of Independent Schools 




83 


85 


83 


National Independent Private Schools Association 




t 


t 


75 


Other private schools 




74 


84 


76 


Schools on list frame by school type: 
Catholic 






88 


83 


Parochial 




— 


87 


83 


Diocesan 




— 


89 


83 


Private order 




— 


90 


84 


Other religions 




— 


79 


75 


Conservative Christian 




— 


77 


70 


Affiliated 




— 


83 


75 


Unaffiliated 




— 


77 


81 


Nonsectarian 




— 


83 


82 


Regular 




— 


84 


83 


Special emphasis 




— 


79 


78 


Special education 




— 


86 


81 



“Not available. (Data were not collected or not reported.) 

*Not applicable. (Not listed as a separate association group.) 

^Basic weight is used to reflect the probability of selection. Rates restricted to schools providing teacher Usts. 
^Round 1 used one Lutheran Church group and one Jewish school group. 

SOURCES: NCES (1991d); Scheuren et al. (1996); Monaco et al. (1998). 





Table 5-11. — Weighted response rates for the Teacher Survey, by selected characteristics and 
school sector; Rounds 1-3 







Weighted response rate (percent)* 






Round 2 (1990-91) 


Round 3 (1993-94) 


Characteristics 


Public 


Private 


Public 


Private 


Region: 






86 


89 


West 


90 


82 


South 


92 


84 


90 


82 


Northeast 


85 


84 


85 


82 


Midwest 


92 


87 


90 


78 


S' 


S' 


P<0.001 


P<0.001 


Urbanicity: 


Central city 


87 


84 


91 


/y 


Urban fringe/large town 


89 


84 


87 


80 


Rural 


93 


85 


91 


83 




s' 


NS^ 


P<0.001 


P=0.009 


School level: 


Combined 


91 


82 


88 


77 


Secondary 


90 


87 


87 


84 


Elementary 


91 


85 


88 


81 


NS’ 


S' 


P=0.523 


P=0.001 


School size: 


1-149 


92 


78 


91 


75 


150-499 


92 


86 


89 


82 


500-749 


90 


84 


88 


82 


750-b 


89 


87 


87 


81 






s' 


P<0.001 


P<0.001 


School type: 


Non-regular 






86 




Regular 






88 

P=0.017 




Minority enrollment: 


<5.5% 


t 




91 




5.5-20.5% 






89 


t 


20.5-50.5% 


t 




88 


t 


>50.5% 


t 




83 

P<0.001 




School sampled with certainty: 


Yes 


. ♦ 




88 




No 


♦ 




88 

P=0.801 




School sampled in Round 2 SASS: 


Yes 


♦ 




88 


88 


No 


♦ 


t 


88 

P=0.427 


82 

P<0.001 


New teacher: 


Yes 


♦ 




90 


81 


No 


♦ 


t 


88 

P=0.003 


80 

P=0.321 


Race of teacher: 


AIAE 


♦ 


t 


87 


78 


API 


♦ 


t 


88 


82 


B lack/Non-Hispanic 


f ■ 


t 


84 


70 


Hispanic 


♦ 


t 


86 


79 


White 


♦ 


t 


90 


68 


Other 


♦ 


t 


81 

P<0.000 


81 

P<0.001 


1991-92 PSS status: 


Respondent 


♦ 




♦ 


81 


Nonrespondent 


■ t 


‘ t 




64 


Not in 1991-92 PSS 


t 


t 


♦ 


68 

P=0.002 



*Not applicable. 

^The basic weight is used to reflect the probability of selection. 

indicates a significant association between response rates for any levels of the variable at the 10 percent significance level. 
^NS indicates there is not a significant association between response rates of the variable at the 10 percent significance level. 
SOURCES: Scheuren et al. (1996); Monaco et al. (1998). 
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Table 5-12. — Percent of items with selected response rates for the Teacher Survey: Rounds 1-3 





Percent of items with 


response rates: 


Minimum of item 


Sector 


> 90 percent 


<75 percent 


response rate 
(percent) 


Round 1 (1987-88) 


Public 


90 


1 


64 


Private 


89 


1 


60 


Round 2 (1990-91) 
Public 


84 


0 


76 


Private 


79 


1 


71 


Round 3 (1993-94) 


Public 


91 


0 


71 


Private 


89 


1 


69 


BIA 


84 


3 


70 



SOURCES; NCES (1991d); Gruber, Rohr, and Fondelier (1993, 1996). 



the item blank. The second item with low response was the space for entering the total of a set 
of responses to items asking teachers to report their hours spent in school, during the most recent 
full week, on five categories of school-related activities. Because of data reporting problems, the 
data for this entire set of items, which also covered time spent on school-related activities after 
school hours, were excluded from the public-use data tapes for the Teacher Survey. 

Most of the low item response rates observed on the Round 2 Teacher Survey questionnaires 
were for items that asked teachers who had answered "Yes" to a question to report a related 
number or amount. For example, in an item on teacher training, several teachers who reported 
that they had taken courses in one or more of the subjects listed failed to enter the number of 
such undergraduate and graduate courses that they had taken (or to check the box for "none" in 
one of these categories). Similarly, teachers who reported that they had received certain types of 
income frequently failed to report the amounts. 

In Round 3, 7 items in the BIA school teacher questionnaire had response rates below 75 percent. 
They included items on time spent working as a teacher at the school; other main activity; the 
number of separate classes (or sections) taught; and salary amount and family income. 

5.7 DATA PROCESSING 

The sequence and nature of the data processing operations for Rounds 1 to 3 of the Teacher 
Survey are similar to those described for the other three basic SASS surveys in chapters 2 to 4. 
Following the same sequence used in the preceding chapters, this section discusses: edit 

procedures (5.7.1), imputation (5.7.2), weighting (5.7.3), and variance estimation (5.7.4). 



5.7.1 Edit procedures 

Clerical edits. As part of the clerical edit, codes were assigned to occupation and industry 
entries for teachers whose prior job had been outside the field of education. One difference for 
the Teacher Survey was that data entry keying was verified for a sample of one-third of the 
questionnaires in Round 2, whereas 100 percent verification was used in Round 3 and for the 

other surveys. 

In some instances, the clerks who reviewed the questionnaires rejected in the computer pe-edit 
operation were required to contact the teachers in an attempt to resolve discrepancies and 
omissions for critical items. The most frequent reasons for pre-edit rejection of questionnaires 
were inconsistencies in responses relating to full-time and part-time employment status and 
failure to respond to a question on main teaching assignment. 

Interview status edit. Following the main computer edit and prior to imputation, an interview 
status edit was performed. Questionnaires were classified as out of scope if, for any reason, they 
were not members of the target population for the Teacher Survey (see section 5.2). 
Questionnaires for in-scope teachers were classified as interviews and included in the tabulations 
only if all the following conditions were met: 

Q The teacher reported the year that he or she started working as an elementary or secondary 
teacher; 

a At least one part of the educational background section had an acceptable response; 

a The teacher reported his or her main assignment field and whether or not he or she was 
certified in that field; 

a The teacher reported at least one grade level of students currently being taught by him or her; 
and 

m There were responses for at least 30 percent of the set of required items that a teacher should 
complete. 

Teachers whose questionnaires did not meet these minimum requirements were treated as 
nonrespondents in computing the unit response rates presented in section 5.6.5. The editing 
procedure changed the preliminary interview status code for 4 percent of interviewed teachers in 
public schools and about 6 percent of interviewed teachers in private and in BIA schools. The 
changes involved reassigning interview cases as noninterview cases, and in some instances, as 
out-of-scope cases. 

Errors uncovered in editing. Reviews of unedited and edited questionnaires and reviews of 
output from processing operations (clerical edit, computer pre-edit, computer edit, and 
imputation) provide indications of measurement error, some systematic and some anecdotal. 
Such findings generally support and extend what has been learned in more formal evaluations 
through reinterviews, cognitive interviews, and record checks (such as the Teacher Transcript 
study) discussed in section 5.6. 



For Round 2, information on both measurement error and item nonresponse is available from a 
review of post-edit item response rates, pre-edit reject rates and edit change tallies (Jenkins, 
1992a). This review, which covered all four of the basic SASS surveys, showed that, in 
comparison with the other surveys, pre-edit reject rates for the Teacher Survey were relatively 
low but that item' nonresponse rates were somewhat higher than for the other surveys. 
Nonresponse was relatively high for several parts of the item on classes in departmentalized 
courses. This item had already been identified, in cognitive interviews and questionnaire 
reviews prior to data processing, as having significant response problems. The edit change 
tallies showed that, as had been observed at earlier stages, respondents frequently answered 
items that did not apply to them. For Round 3, the Data File User's Manual (Gruber et al., 1996) 
lists the data items and the counts of edit changes for each item in the teacher questionnaires 
(public, private, and BIA) used in that round. 

5.7.2 Imputation 

Imputation procedures for the Teacher Survey followed the same general pattern as imputation 
for the School Survey (section 2.7.3). Some responses were assigned for missing or inconsistent 
values during the computer edit and, in a few cases where the correct entry was obvious, items 
were changed without contacting respondents in the initial clerical edit and in the resolution of 
rejected cases from the computer pre-edit. However, in Rounds 2 and 3 most of the imputation 
was done in a computer operation following the computer edit. 

In Round 1 no computer imputation was done initially for missing or blanked values and the 
values were left blanks. Item imputation for Round 1 was complete when the data were released 
on a CD-ROM with data from Rounds 1 through 3 (NCES, 1998). In Round 2, computer 
imputation proceeded in two stages. In the first stage, missing or previously blanked values for 
selected items were imputed by using other information for the same teacher or making 
assumptions about the respondent's intended answer, for example, that not answering a question 
implied a response of "No." In the second stage, a hot deck procedure was used to impute the 
remaining missing values. The matching variables used to form imputation groups for each item 
and the order of their collapsing (when necessary to form sufficiently large imputation groups) 
are described in Part VUI of the Data File User's Manual (Gruber et al., 1993, for Round 2; and 
Gruber et al., 1996, for Round 3). In Round 3, a third stage of clerical imputation was added to 
handle items for which there were very few missing responses and a few cases where no hot 
deck donor was available. 

In Round 1 no flags were assigned to identify items imputed during the computer edit or earlier 
stages of processing. In Round 2, flags were assigned to all items imputed in the computer 
imputation operation. Those imputed in the first stage were flagged as "internal imputation" and 
those imputed in the second stage were flagged as "donor-based" imputation. Items imputed 
prior to the computer imputation were not flagged. In Round 3, flags distinguished between the 
stage or type of imputation: rational adjustment to the original entry, other internal imputation, 
donor-based imputation, and clerical imputation. 



5.7.3 Weighting 



The overall weights for teachers were developed in the same general way for each round. In 
Rounds 2 and 3 the weights were the products of six components; in Round 1 the last of these 
components was not used. The components are as follows: 

■ The basic sampling weight was the inverse of the teacher's overall selection probability, that 
is, the product of the school selection probability and the probability of selecting the sample 
teacher within the school. 

■ The sample adjustment factor accounted for unusual circumstances, such as mergers, splits or 
duplications, that had affected the school's probability of selection. 

■ The school nonresponse adjustment factor was designed to account for schools that did not 
provide teacher lists for sampling. 

■ The teacher nonresponse adjustment factor was designed to account for sample teachers for 
whom acceptable questionnaires were not obtained. 

■ The frame ratio adjustment factor was designed to reduce sampling error by adjusting 
sample estimates based on frame counts of teachers in sample schools to agree with the 
corresponding frame counts based on data for all schools. 

■ The teacher adjustment factor was used to force agreement between estimates of total 
number of teachers based on the School and Teacher Survey questionnaires. This adjustment 
factor was not used in Round 1. 

Each of the last four factors was computed and applied within weighting cells comprised of 
schools or teachers with similar characteristics. Detailed descriptions of the weighting cells and 
the rules for collapsing them when necessary are provided in the Sample Design and Estimation 
Reports for each round (Kaufman, 1991, for Round 1; Kaufman and Huang, 1993, for Round 2; 
and Abramson et al., 1996, for Round 3). In Round 3, the frame ratio adjustment factor was not 
applied in BIA schools since teacher data were not available from the BIA school frame. 

In a review of the teacher weights for Round 2, some of the CCD teacher counts used in the 
numerator of the frame ratio adjustment factor were found to be one-tenth of the correct values, 
possibly as a result of data keying errors. The problem was most severe in Iowa, and the teacher 
weights for that state were recomputed. The problem may have existed in other states, but, given 
the difficulty of identifying such cases and the late stage at which the problem was discovered, 
no other corrections were made (Kaufman and Huang, 1993). 

Table 5-13 shows the median and the range of the teacher adjustment factors for both public and 
private school teachers. The maximum and minimum values for these adjustment factors were 
constrained by collapsing rules, which required that any cell with a factor outside the range from 
0.667 to 1.500 be collapsed with another cell according to prescribed rules. For the majority of 
cells, the teacher adjustment factors were greater than 1.000, indicating that teacher counts 
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Table 5-13. — Adjustment factors applied to the teacher sampling weights, by sector: Round 2 



Sector 



Maximum 



Minimum 



Median 



Public 

Private 



1.498 

1.478 



0.711 

0.850 



1.074 

1.153 



SOURCE; National Center for Education Statistics, SASS, Teacher Survey (1990-91). 

reported in the School Survey exceeded the number of teachers included on the teacher listing 
forms for the same schools. Section 5.4.3 above discussed the studies that examined the 
discrepant teacher counts reported by the same school. 

5.7.4 Variance estimation 

For public school teachers, table 5-14 shows the estimates and their coefficients of variation 
(CVs) by state of the percent of teachers whose highest degree earned is a bachelor's degree, 
percent of teachers with less than 3 years of teaching experience, and the average total earned 
income in Round 3. 

Overall 52 percent of public school teachers have a bachelor's degree as the highest degree 
earned, with a considerable variability across states from 20 percent in Connecticut to 79 percent 
in North Dakota. The CVs for the state estimates are generally under 5 percent, and all are less 
than 10 percent. Several of the larger CVs occur because of the small percent of teachers with a 
bachelor's degree as the highest degree earned in that state. With an overall rate of only 10 
percent of teachers having less than 3 years of full-time teaching experience, the CVs for the 
state estimates for this percent are much larger, with two in five of them exceeding 10 percent. 
The CVs for average total earned income are all very small, with the largest being 1.7 percent. 

Table 5-15 shows the corresponding estimates and CVs for private schools teachers by private 
school types. The CVs for the percent of teachers whose highest earned degree is a bachelor's 
degree are less than 5 percent for all school types except for special education schools. The 
percent of teachers with less than 3 years of full-time teaching experience is higher for private 
schools than public schools, a feature that tends to reduce the CVs. The CVs for this estimate for 
the various types of private schools are all less than 10 percent except for other religious 
unaffiliated schools and special education schools. The CVs for average total earned income are 
less than 5 percent except for other religious unaffiliated schools. 

Generalized variance functions. Generalized variance functions (GVFs) provide a simple means 
to approximate sampling errors associated with survey estimates. They are useful for analysts 
who do not work with microdata files or lack the software for computing sampling errors for 
complex sample designs. For Round 2 of the Teacher Survey, Salvucci et al. (1995) provide 
GVFs for estimating CVs of teacher totals (e.g., number of male teachers), averages (e.g., 
average number of mathematics or computer science courses taken), and proportions (e.g., 
proportion of married teachers). They examined alternative GVF models for estimating CVs of 
estimates from the Teacher Survey and concluded that the model CV=[A-I-5(X)]'^' gave on 



Table 5-14. — Coefficients of variation (CVs) for selected items for public school teachers, by 
state: Round 3 



Highest degree earned 
State was bachelor's degree 


Less than 3 years 
CV of fulLtime teaching 


CV 


Average total 
earned income 


CV 




Percent 


Percent 


Percent 


Percent 


Dollar 


Percent 


Total 


52 


0,6 


10 


2,0 


$35,228 


0.3 


Alabama 


39 


4.9 


12 


6.6 


28,062 


0.7 


Alaska 


59 


1,8 


8 


8,0 


46,903 


0,4 


Arizona 


51 


3,5 


14 


7,2 


32,380 


0,8 


Arkansas 


65 


3,2 


9 


9,1 


27,016 


0,7 


California 


59 


3,0 


11 


7,9 


40,898 


0,9 


Colorado 


47. 


3,6 


10 


8,7 


33,118 


1,0 


Connecticut 


20 


5,8 


6 


11.3 


48,905 


0.7 


Delaware 


46 


3.5 


9 


11,5 


38,671 


1.2 


District of Columbia 


41 


4,3 


12 


12.8 


43,108 


1.4 


Florida 


57 


2.0 


9 


10,7 


32,389 


0.9 


Georgia 


49 


3.0 


13 


8,5 


29,705 


0.6 


Hawaii 


48 


4.3 


15 


9.8 


35,920 


1,1 


Idaho 


74 


2.1 


10 


11,2 


27,420 


1.1 


Illinois 


50 


2.7 


9 


6,9 


37,814 


1,3 


Indiana 


21 


8.1 


6 


14.1 


36,695 


1.0 


Iowa 


61 


3.0 


9 


12,6 


28,351 


1.3 


Kansas 


54 


2.6 


11 


6.9 


30,501 


0,8 


Kentucky 


23 


9.3 


9 


15,2 


31,519 


0.9 


Louisiana 


61 


2.6 


10 


8,4 


24,923 


0.6 


Maine 


68 


2.8 


6 


15,4 


30,649 


1.0 


Maryland 


43 


4,4 


13 


7,6 


39,350 


0.9 


Massachusetts 


39 


3.2 


7 


8,6 


38,340 


0,5 


Michigan 


47 


4.0 


5 


19,6 


44,055 


1.1 


Minnesota 


63 


3.2 


10 


10,3 


35,999 


1,1 


Mississippi 


56 


2.7 


10 


8,5 


24,990 


0,6 


Missouri 


54 


3.7 


9 


10,1 


29,216 


1,6 


Montana 


71 


1.9 


10 


6,2 


27,591 


0.8 


Nebraska 


62 


2.7 


7 


9.4 


26,859 


1.7 


Nevada 


51 


4.4 


12 


8.5 


34,515 


1,1 


New Hampshire 


60 


3,3 


7 


12,4 


34,159 


1.2 


New Jersey 


56 


5.1 


7 


14,4 . 


46,735 


1,6 


New Mexico 


53 


3,0 


13 


8,3 


27,513 


0,7 


New York 


25 


7,6 


11 


11.7 


47,016 


1.7 


North Carolina 


62 


2,4 


9 


9,4 


28,005 


0.7 


North Dakota 


79 


1.4 


9 


11.4 


24,757 


1.0 


Ohio 


53 


4.1 


7 


11.7 


34,814 


1,1 


Oklahoma 


57 


3.2 


11 


8,3 


27,646 


0,6 


Oregon 


52 


3.8 


6 


14,7 


34,893 


1.1 


Pennsylvania 


47 


4,9 


7 


15,6 


41,844 


1.0 


Rhode Island 


40 


6,2 


6 


15,7 


40,789 


0.7 


South Carolina 


49 


5.2 


11 


12,6 


29,112 


0.9 


South Dakota ‘ 


75 


1.6 


8 


8.0 


24,430 


0.9 


Tennessee 


51 


3.9 


13 


9.5 


29,134 


0,9 


Texas 


70 


1.9 


12 


7.1 


29,176 


0,8 


Utah 


71 


2.0 


11 


5.6 


28,907 


0.7 


Vermont 


49 


3.6 


8 


14.2 


34,016 


1.3 


Virginia 


64 


3.2 


10 


11,7 


31,880 


1,0 


Washington 


56 


3.4 


10 


9.9 


37,152 


1.0 


West Virginia 


42 


4.0 


3 


20.6 


30,366 


0,5 


Wisconsin 


59 


2.8 


8 


12.5 


36,448 


0,9 


Wyoming 


71 


1.3 


7 


8.9 


30,005 


0.8 



SOURCE: CVs are computed using estimates and standard errors of estimates from Bobbitt, Broughman, and Gruber ( 1 995), 
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Table 5-15. — Coefficients of variation (CVs) for selected items, for private school teachers, by 
school type: Round 3 , 



Highest degree earned 
School tvoe was Bachelor's degree 


Less than 3 years of full- : 
CV time teaching experience 


CV 


Average total 
earned income 


CV 




Percent Percent 


Percent 


Percent 


Dollar Percent 


All private teachers 


59 


1.1 


20 


2.8 


$22,739 


0.9 


Catholic 


64 


1.1 


17 


3.4 


22,356 


0.8 


Parochial 


71 


1.2 


18 


4.7 


20,058 


0.9 


Diocesan 


62 


2.2 


16 


4.8 


22,720 


1.1 


Private order 


45 


3.5 


18 


9.5 


28,426 


1.8 


Other religious 


58 


1.9 


24 


5.1 


20,440 


2.1 


Conservative Christian 


66 


2.6 


34 


6.2 


16,839 


2.2 


Affiliated 


55 


2.9 


19 


6.8 


23,995 


1.6 


Unaffiliated 


51 


4.8 


18 


12.7 


20,471 


5.8 


Nonsectarian 


53 


2.7 


20 


5.7 


26,712 


1.5 


Regular 


52 


3.2 


17 


7.2 


26,768 


2.0 


Special emphasis 


55 


3.9 


28 


8.6 


25,431 


2.7 


Special education 


54 


7.8 


22 


17.8 


27,763 


3.4 



SOURCE: CVs are computed using estimates and standard errors of estimates from Bobbitt, Broughman, and Gruber (1995). 



average the best fit. The parameters of the GVFs (i.e., the values of A and B in the model) are 
provided by sector, region, region within sector, minority status within sector, and state (for 
public school teachers). 

Variance estimation of imputed survey data. Zhang, Brick, Kaufman, and Walter (1998) 
conducted an empirical study of variance estimation taking account of the imputations in the 
Round 3 teacher questionnaires. They applied Shao and Sitter's (1996) bootstrap variance 
estimation method and found that when the stage 2 donor-based imputation rate is high (above 
10 percent), the increase in the estimated standard error can be large compared with the standard 
error computed by treating the imputed values as reported values, especially for categorical 
variables. When the imputation rate is low, the increase in standard error is not severe. In 
Round 3, there were relatively few items that had high stage 2 imputation rates (only 1 1 out of 
249 items from the public school teacher questionnaire). The authors suggest that data users 
might be provided with standard error estimates that take account of the imputed values for 
estimates of totals and means; users could then use these standard error estimates to develop 
adjustment factors when estimating standard errors of other estimates. 

5.8 EVALUATION OF ESTIMATES 

This section describes comparisons of Teacher Survey estimates with data from other sources, 
including the School Survey, the CCD, and to a lirmted extent, data available from other sources. 
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Some of these comparisons were made as part of prepublication reviews; others were made 
subsequent to publication. 

Round revaluations. Prior to publication of Round 1 results, estimates of full-time equivalent 
(FTE) teachers by state based on the Teacher Survey were compared with estimates from the 
School Survey. As noted earlier, for most states the Teacher Survey estimates were lower, 
reflecting the fact that the number of teachers listed by the school on the form used to sample 
teachers was often less than the count of teachers reported for the same school in the School 
Survey. In addition, there was evidence that many schools provided counts of FTE teachers that 
were too high. In the average state, 19 percent of the schools with some part-time teachers 
reported the same counts for total and FTE teachers (Choy, Medrich, Henke, and Bobbitt, 1992). 
The extent of this phenomenon varied by state, from 10 percent in Alaska and Hawaii to 31 
percent in Colorado (Kaufman, 1990). Consideration was given to the possibility of adjusting 
the Teacher Survey estimates to force agreement with CCD counts, but such an adjustment 
would not have resolved the discrepancies between the FTE teacher counts from the School and 
Teacher Surveys. 

0 

The Round 1 Teacher Survey estimate of average hours spent' in a week on school and school- 
related activities (40.3 hours) proved to be substantially below the corresponding estimate from 
the 1985 Public School Survey (50.4 hours). There were several differences between the two 
surveys in the questionnaire items used to produce these two estimates. In addition, the Round 1 
Teacher Survey estimates may have been low because there was no imputation when responses 
were provided for some but not all of the relevant items. On the basis of this comparison, it was 
decided that no data on this topic would be published or included in the public-use microdata 
files (Hammer, 1990). 

Teacher Survey estimates of salary were compared with data available from private 
organizations. The Teacher Survey estimate of average base salary for public school teachers 
was $26,231, somewhat below the average salary ($28,071) reported by the American Federation 
of Teachers (Nelson, 1990) and the National Education Association (1990). The higher figures 
from the latter two sources are believed to result from the inclusion, by some states, of other 
kinds of instructional expenditures in the category that covers teachers' salaries (Choy et al., 
1992; Fowler, 1990). 

Round 2 evaluation of estimates. The use of a teacher adjustment factor as one component of the 
Teacher Survey weights for Round 2 guaranteed that teacher estimates from the Teacher and 
School Surveys would agree for each of the weighting cells. However, as described in more 
detail in section 2.7, comparisons of School Survey public school FTE teacher estimates with 
counts from the CCD showed that the School Survey estimates for nine states were at least 15 
percent higher than the CCD counts. Investigation of this problem led to a series of additional 
processing and reweighting steps to make SASS state estimates of the number of public schools 
more consistent with the CCD counts and to make enrollment and teacher count data consistent 
with the CCD on a school-by-school basis. These steps included reclassifying as out of scope 
teacher file records for teachers who taught only grades that were no longer considered part of a 
sample school and reinstating records for teachers at sample schools previously classified as out 
of scope because of apparent mergers (Fondelier, 1992). 
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Final School Survey estimates, by state, of FTE teachers were compared with counts from the 
1990-91 CCD. At the national level, the SASS estimates exceeded the CCD count by 2.8 
percent. The SASS estimates for South Dakota and Wisconsin were 31.7 percent and 17.1 
percent, respectively, above the CCD counts for those states. For 9 states, differences were in 
the range from 10 to 15 percent, with SASS being higher in all 9 states. For all other states, 
differences were less than 10 percent. The SASS and CCD data by state are shown in chapter 
XII of the Round 2 Data File User's Manual (Gruber, Rohr, and Fondelier, 1993). 

Round 3 evaluation of estimates. In Round 3 a comparison was made by state between the 
number of public teachers estimated from the Teacher Survey and the CCD State Nonfiscal 
Survey (see table 5-16). This comparison was conducted to spot gross errors since there are 
several reasons why the number of teachers, in FTE counts, estimated from the Teacher Survey 
would differ from CCD's State Nonfiscal Survey counts. The CCD counts are statewide official 
tallies of teachers, reported from a central agency, and unduplicated to account for teachers in 
multiple districts or schools. The Teacher Survey depends in part upon the cooperation of the 
schools to provide a list of all teachers and the survey population is constrained by the 
operational procedures for the survey (see section 5.2). For example, if a teacher is out on 
maternity leave or has taken another job in some other school when sampled, the questionnaire is 
declared Out of scope, since the designated teacher is not available; however, from the state's 
point of view, there is still a teaching position at the sampled school. Also, when a sample 
school is declared out of scope, such as for merging with another school that is not in sample, the 
teachers that were selected for sample are also out of scope. While such factors affect relatively 
small proportions of the sampled cases, there may be a cumulative effect on the overall count of 
teachers in some states. 

At the national level, the Teacher Survey estimate of the number of FTE teachers was slightly 
more than 2 percent lower overall than the CCD count. In the following 1 1 states, the Teacher 
Survey estimate was at least 4.5 percent higher than the CCD count: Alaska, Arkansas, 

Delaware, Hawaii, Iowa, Kentucky, Missouri, Montana, Pennsylvania, Wisconsin, and 
Wyoming. In 6 states — Alaska, Arkansas, Kentucky, Montana, Pennsylvania, and Wisconsin — 
the Teacher Survey estimate of FTE teachers was higher than the CCD count. SASS and CCD 
data by state are shown in table XII-9 of the Round 3 Data File User's Manual (Gruber, Rohr, 
and Fondelier, 1996). 
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Table 5-16. — Full-time equivalent (FTE) teachers in public schools from the Round 3 SASS 
Teacher Survey and the 1993-94 Common Core of Data (CCD) State Nonfiscal 
Survey, by state 



State 


FTE teachers 
CCD 


FTE teachers 
SASS 


SASS/CCD (percent) 


50 states and D.C. 


2,505,074 


2,452,057 


98 


Alabama 


43,002 


43,660 


102 


Alaska 


7,193 


7,806 


109 


Arizona 


37,493 


35,779 


95 


Arkansas 


26,014 


29,698 


114 


California 


221,779 


203,711 


92 


Colorado 


33,661 


33,809 


100 


Connecticut 


34,526 


33,438 


97 


Delaware 


6,380 


6,841 


107 


District of Columbia 


6,056 


5,149 


85 


Florida 


110,653 


104,225 


94 


Georgia 


75,602 


72,763 


96 


Hawaii 


10,111 


10,609 


105 


Idaho 


12,007 


11,467 


96 


Illinois 


110,874 


106,172 


96 


Indiana 


55,107 


55,170 


100 


Iowa 


31,616 


33,159 


105 


Kansas 


30,283 


28,629 


95 


Kentucky 


37,324 


40,002 


107 


Louisiana 


46,913 


47,649 


102 


Maine 


15,344 


14,478 


94 


Maryland 


44,171 


4U52 


93 


Massachusetts 


58,766 


53,407 


91 


Michigan 


80,267 


79,388 


99 


Minnesota 


46,956 


41,081 


87 


Mississippi 


28,376 


29,172 


103 


Missouri 


54,543 - 


59,577 


109 


Montana 


9,950 


12,106 


122 


Nebraska 


19,552 


18,870 


97 


Nevada 


12,579 


12,444 


99 


New Hampshire 


11,972 


11,331 


95 


New Jersey 


84,564 


80,459 


95 


New Mexico 


18,404 


18,742 


102 


New York 


179,413 


171,571 


96 


North Carolina 


69,421 


68,437 


99 


North Dakota 


7,755 


7,564 


98 


Ohio 


107,444 


106,832 


99 


Oklahoma 


39,031 


40,699 


104 


Oregon 


26,488 


23,871 


90 


Pennsylvania 


101,301 


108,074 


107 


Rhode Island 


9,823 


8,565 


87 


South Carolina 


38,620 


38,599 


100 


South Dakota 


9,557 


9,931 


104 


Tennessee 


46,066 


46,081 


100 


Texas 


224,830 


219,133 


97 


Utah 


19,053 


19,038 


100 


Vermont 


8,102 


6,692 


83 


Virginia 


70,220 


62,374 


89 


Washington 


45,524 


45,906 . 


101 


West Virginia 


21,029 


20,251 


96 


Wisconsin 


52,822 


59,437 


113 


Wyoming 


6,537 


7,059 


108 



SOURCE: Gruber, Rohr, and Fondelier (1996). 
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6. THE TEACHER FOLLOWUP SURVEY 



6.1 INTRODUCTION 

The Schools and Staffing Surveys’ (SASS) Teacher Followup Survey (TFS) is conducted in the 
school year following the basic SASS surveys (i.e., Round 1 in 1988-89, Round 2 in 1991-92 and 
Round 3 in 1994-95). Information is collected from a subset of the. teachers who responded to 
the Teacher Survey in the base school year (i.e., Round 1 in 1987-88, Round 2 in 1990-91 and 
Round 3 in 1993-94). On the basis of inquiries to their schools early in the following school 
year, teachers who responded in the base year are classified into three categories: Leavers, those 
who left the teaching profession between the base year and the following year; Movers, those 
who moved to a different school between the base year and the following year; and Stayers, 
those teaching in the same school in both years. The stayers and movers are referred to 
collectively as current teachers and the leavers as former teachers. The sample for the TFS 
consists of all of the former teachers and a subset of the current teachers, with a different 
questionnaire administered to teachers in each of these two groups. 

The main purposes of the TFS Survey are the following: 

■ Measure attrition rates for elementary and secondary teachers, by type of school and by state; 

■ Determine and compare the characteristics and attitudes of leavers, movers, and stayers; 

■ Determine the current economic activities of leavers; 

■ Collect data on attitudes about the teaching profession and job satisfaction; and 

■ Obtain data on educational activities and future plans for all groups. 

This chapter has the same organization as preceding chapters. It has sections covering the main 
phases of the survey operations: sampling frames (6.2), sample design (6.3), content (6.4), data 
collection procedures (6.5), data processing (6.6), measurement error (6.7), nonresponse (6.8), 
and estimation (6.9). Because of the lack of data availability for comparisons, there is no section 
on evaluation of estimates. 

6.2 SAMPLING FRAMES 

The TFS has the same target population as the Teacher Survey — that is, regular full-time or part- 
time teachers during the base school year whose primary assignment was teaching in 
kindergarten or any of grades 1 to 12 in a school that was eligible for the School Survey. Also 
included are persons who, in the base year, were substitutes filling the role of a regular teacher 
on a long-term basis or itinerant teachers (those teaching regularly in more than one school); 
such persons were also included in the Teacher Survey. Details about the selection of the sample 
of schools for the School Survey and the sample of teachers from those schools for the Teacher 
Survey can be found in chapters 2 and 5, respectively. 



The TFS sampling frame consisted of all eligible teachers who had responded to the Teacher 
Survey conducted in the previous year. Teachers who had not responded to the Teacher Survey 
were excluded because of the absence of base-year information for them, thus limiting the utility 
of their TFS responses for analysis. 

Evaluation of the sampling frame. The overall coverage of the target population for the TFS 
depends in large part on the completeness of coverage of the frames used for the base-year 
School and Teacher Surveys. Evaluation of those frames is discussed in chapters 2 and 5, 
respectively. The proportion of the target population for the TFS covered by its sampling frame 
was further reduced by the exclusion of teachers in schools not providing teacher lists for the 
Teacher Survey and by nonrespondents to the Teacher Survey. The loss from schools not 
sending in teacher lists was fairly constant across rounds for both public and private schools, at 
between 4 and 5 percent for public schools and at least double that for private schools. The loss 
due to teacher nonresponse in schools that sent in teacher lists was larger, at about 10 percent for 
the public sector and about 20 percent for the private sector. 

6.3 SAMPLE DESIGN 

A primary sample design objective for the TFS is to support comparative analyses of stayers, 
movers, and leavers for teachers classified by sector (public and private), level (elementary and 
secondary), and years in teaching (new and experienced). Large majorities of teachers in all 
categories are stayers. Consequently, movers and leavers have been oversampled in order to 
reduce the sampling errors of estimated differences among groups. As mentioned in section 
5.4.2, in all rounds new teachers in private schools were oversampled for the Teacher Survey to 
ensure a sufficient sample of teachers in this category for the TFS. 

In order to achieve the above design objective, the sample of respondents to the Teacher Survey 
was grouped into 24 strata formed by the cross-classification of school sector (public or private), 
level (elementary or secondary), teaching experience (new or experienced), and status (leaver, 
mover, or stayer). Apart from status, all these variables were available from the Teacher Survey. 

Information to classify responding teachers in the Teacher Survey as leavers, movers, or stayers 
was obtained by asking school principals for the teachers' current occupational status. This 
classification activity was initiated with an advance letter being sent to school districts in late 
summer/early fall in all rounds. This letter thanked the district for its participation in SASS in 
the previous year, introduced the TFS and its importance, and notified the district that the U.S. 
Census Bureau would soon be contacting schools in the previous year's survey for the current 
employment status of the sampled teachers. School principals were then sent a Teacher Status 
Form on which they (or other knowledgeable school staff members) were asked to report for 
each teacher who had participated in the Teacher Survey whether the teacher was still at the 
school in a teaching or nonteaching capacity, had left the school to teach elsewhere, or had left 
the school for a nonteaching occupation. The Teacher Status Form was mailed to principals in 
the fall of the year following the Teacher Survey (in October, 1988, for Round 1; in September, 
1991, for Round 2; and in September, 1994, for Round 3). In Round 3, reminder postcards were 
sent to schools shortly after the Teacher Status Form mailing. In all rounds, schools that had not 
responded within about one month were telephoned to collect the information requested for the 
teachers listed on the forms. If a school did not respond, all its sampled teachers were classified 





as stayers, since that is the predominant status. In the few cases where a school responded but 
did not specify a teacher's current status, the teacher was classified as a leaver for sample 
selection purposes. Since leavers were sampled at the highest rate, this procedure ensured that 
such teachers were sampled at a rate not lower than the rate of the group to which they truly 
belonged. 

Sample allocation. For all three rounds, the sample size for the TFS was about 7,200 teachers, 
consisting of about 5,100 public school teachers and 2,100 private school teachers. These totals 
were allocated among the 24 strata defined above in a way that produced a sufficient sample of 
teachers in each stratum to permit comparisons across strata (for example, to compare the 
proportion of former teachers among new elementary school teachers in public schools with the 
corresponding proportion in private schools). The allocation to all 24 strata can be found in the 
TFS User Manuals or technical notes of other publications for each of the three rounds (Bobbitt, 
Faupel, and Bums, 1991; Whitener, Kaufman, Rohr, Bynum, and King, 1994; Whitener, Gruber, 
Rohr, and Fondelier, 1999). Rounds 2 and 3 used exactly the same allocation. Thus, for 
example, in these rounds the allocation yielded overall about 2,400 new and 4,800 experienced 
teachers, and about 2,300 former and 4,900 current teachers. Round 1 selected more former 
teachers (about 3,000) and correspondingly fewer current teachers (about 4,200). 

Sample selection. For all of the leaver strata and some of the mover strata, it was necessary to 
include all Teacher Survey respondents in the TFS sample in all three rounds. In each of the 
strata for which a subsample of the Teacher Survey respondents was to be selected, the 
responding sample teachers were sorted in a specified order before the selection was made. The 
sorting variables and their order of application changed from year to year for both sectors, as 
shown in table 6-1. The samples for the TFS were then selected by systematically sampling 
within strata, with the ordering of the lists giving the benefit of implicit stratification. 



Table 6-1. — Variables used in ordering teacher lists, by sector: Rounds 1-3 



Round 1 


Round 2 


Round 3 


(1988-1989) 


(1991-92) 


(1994-95) 


Public 


Private 


Public 


Private 


Public 


Private 


Census region 


Association 


Teacher subject 


Teacher subject 


Teacher subject 


Teacher subject 


Urbanicity 


Urbanicity 


Census region 


Affiliation 


Census region 


Association/ 

Affiliation^ 


Teacher subject^ 


Teacher subject 


Urbanicity 


Urbanicity 


Urbanicity 


Urbanicity 


School 


School 


School 


School 


School 


School 


enrollment 


enrollment 


enrollment 


enrollment 


enrollment 


enrollment 






SASS teacher 


SASS teacher 


SASS teacher 


SASS teacher 






control number 


control number 


control number 


control number 



'Association group for the list frame, affiliation group for the area frame. 

^For elementary teachers, teacher subject refers to general elementary, special education, or other. For secondary teachers, teacher subject refers 
to math, science, English, social studies, vocational education, special education, or other. 

SOURCES: Bobbitt et al. (1 99 1 ); Whitener et al. (1994); Whitener et al. (1999). 
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The teachers were sampled within strata with unequal selection probabilities, with the aim of 
making the TFS sample in each stratum more nearly self-weighting than the Teacher Survey 
sample. In all three rounds, teachers were sampled by probability proportional to size sampling, 
but the measure of size changed somewhat across rounds: For Round 1 the measure of size was 
the basic weight for the Teacher Survey sample (i.e., the inverse of the selection probability); in 
Round 2, it was the intermediate teacher weight that included the sampling adjustment factor, the 
school nonresponse adjustment factor, and the ratio adjustment factor (see section 5.7.3); in 
Round 3, it also incorporated the teacher noninterview factor. The measure of size changed for 
each round depending on the status of the SASS teacher weighting process for each round. At 
the time the TFS sample had to be selected, as many as possible of the SASS teacher weighting 
factors that were available were used in this measure of size and this varied for each round. 

In some cases the status of teachers, as determined from their TFS responses, differed from the 
status reported for them on the Teacher Status Forms that were used in sampling. Such 
differences could result from changes in status during the school year, frorn reporting errors on 
the Teacher Status Forms, from an incorrect assignment of stayer or leaver status to teachers 
whose current status was unknown, or from reporting errors on the TFS questionnaires. 
Differences of the first three kinds do not bias the survey estimates, but they lead to a less 
efficient sample. In particular, a problem arises when leavers or movers are incorrectly classified 
as stayers, in which event they receive basic weights substantially greater than those of other 
teachers in their categories. Incorrect reporting of status on the TFS questionnaire, of course, 
causes bias in the survey estimates. 

6.4 CONTENT 

The TFS questionnaires (one for current teachers and one for former teachers) collect 
information to measure attrition rates; to compare leavers, movers, and stayers; to determine the 
current economic activities of leavers; to obtain data on educational activities and future plans 
for all groups; and to collect data on attitudes about the teaching profession and job satisfaction. 
It has been suggested that the TFS provides the best national database available for monitoring 
year-to-year flows of teachers, as well as a vehicle for tracking actual teacher career transitions 
(as distinct from teacher reports of activities in the prior year, and plans for the coming year) 
(Ingersoll, 1995; Boe, 1996). 



The subjects covered by the current and former teacher questionnaires are as follows (Salvucci et 
al., 1997): 



Current Teacher Questionnaire: 



Former Teacher Questionnaire: 



■ Primary occupational status 



■ Primary occupational status (full time) 



(full time, part time) 

■ Primary teaching assignment by field 

■ Teaching certificate 

■ Level of students taught 

■ School community type 

■ Expected duration in teaching 



■ Type of business 

■ Primary activity 

■ Time planning to spend in current job 

■ Reasons for leaving teaching 

■ Plans for returning to teaching 



Current Teacher Questionnaire (continued): 

■ Reasons for leaving previous school 
(movers only) 

■ Level of satisfaction 

■ Possible areas of dissatisfaction 

■ New degrees earned or pursued 

■ Marital status 

■ Number of children 

■ Academic year base salary 

■ Combined family income 

The TFS questionnaires for all rounds also asked for information that would facilitate further 
recontacts with the sample teacher. This information was requested because of the interest in 
developing a longitudinal study of teachers' careers. Singer and Willett (1996) discuss 
methodological issues in the design of a longitudinal study of teachers. They recommended a 
followup period for beginning teachers of at least 12 years with at least 6 equally spaced waves 
of data collection. NCES was also encouraged to consider additional brief annual contacts to 
ensure that essential data on the intervening years could be reliably obtained. However, there 
have been no further recontacts of these TFS samples. 

Content changes. Some changes have been made to the wording and order of specific items 
from one round to the next. Additionally, in Round 2, for items common to both the TFS and the 
Teacher Survey, the wording of the TFS item was changed to be consistent with the Teacher 
Survey item. In Round 3, there were very few items asked on both surveys so this practice was 
minimized. In Round 3, a new section consisting of about 20 items was added to the current 
teacher questionnaire to collect data on teaching methods. Using data from these new questions, 
a report by Henke, Chen, Goldman, Rollefson, and Gruber (1999) examined teachers' practices 
in four areas of instruction: the roles that teachers and students play in learning activities, the 
materials and technology used in the classroom, the kinds of learning tasks that students do both 
in the classroom and at home, and how teachers assess and evaluate student learning. 

6.5 DATA COLLECTION PROCEDURES 

Data collection for the TFS was initially by mail, with the questionnaires being sent to the 
sampled teachers' home addresses. In preparation for this mailing, the Teacher Survey 
questionnaire collected the home addresses for the teachers in that survey, together with contact 
information for two persons who would know how and where to get in touch with the teachers. 
Teacher Followup questionnaires were mailed to sampled teachers in March 1989 for Round 1, 
in January 1992 for Round 2, and in January 1995 for Round 3 (Bobbitt et al., 1991; Whitener et 
al., 1994; Whitener et al., 1999). In all rounds, teachers who were sent questionnaires that were 
inappropriate for their status (current or former teachers) were asked to return them so that the 
correct version could be sent to them. A second questionnaire was mailed about 4 to 5 weeks 
later to teachers who had not responded by that time. In Round 3, only 34 percent of stayers and 
19 percent of leavers had responded to the initial mailing before the second mailing occurred. 
The final mail return rates were 44 percent for stayers, and 28 percent for leavers (Cole, Parmer 
and Schwanz, 1997). 



Former Teacher Questionnaire (continued): 

■ Possible areas of dissatisfaction 

■ New degrees earned by type and field 

■ Marital status 

■ Number of children 

■ Salary 

■ Combined family income 
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In all three rounds, nonrespondents to the mail questionnaires were interviewed by telephone by 
U.S. Census Bureau field representatives in regional offices using paper-and-pencil instruments. 
A single version of the questionnaire designed to accommodate both current and former teachers 
was used (Faupel, Bobbitt and Friedrichs, 1992; Whitener et al., 1994; Whitener et al., 1999). 
Telephone interviewing took place in May-July, 1989 for Round 1, April-May, 1992 for Round 
2, and March-May, 1995 for Round 3, with the earlier dates for later waves reflecting a desire to 
initiate followup as early as possible. The interviewers also tried to contact teachers for whom 
questionnaires had not been mailed because of an incomplete mailing address or because no 
current mailing address had been obtained. If interviewers were unable to contact a sample 
teacher through a contact person or through the use of telephone directory assistance, calls were 
then placed to the school where the teacher had been teaching in the base year in an attempt to 
collect contact information. 

In Round 3, responses were obtained from about 89 percent (unweighted) of the teachers 
sampled for the TFS. Given the low level of response to the mailout effort, this rate indicates the 
importance of and the success achieved by the telephone followup efforts. 

6.6 DATA PROCESSING 

Data processing procedures for all three rounds of the TFS were similar to those used in the basic 
SASS surveys. The main steps were a general clerical edit to check for out of range values, data 
keying, computer pre-edit (range checks on priority items and consistency checks between 
selected items), a review and correction of rejects from the computer pre-edit (only in the first 
two rounds), a computer edit, and the assignment of a final status to the questionnaire. The 
computer edit included range checks, inter-item consistency checks and a blanking operation to 
eliminate items that respondents answered unnecessarily because they did not follow skip 
instructions correctly. In Round 2, 9 percent of the 4,818 stayer questionnaires and 23 percent of 
the 2,014 leaver questionnaires failed the computer pre-edit. Clerks reviewed the questionnaires 
to verify the data keying accuracy and attempted to resolve the reject items. For certain cases, 
the clerk was required to call the respondent to resolve critical inconsistencies or to obtain 
critical data. 

There were some exceptions to the above flow of data processing in Round 3. Within the 
clerical edit, former teacher questionnaires were also assigned industry and occupation codes to 
reflect the respondent's current job. Also, for current teacher respondents teaching in a new state, 
a Federal Information Processing Standards (FIPS) code was assigned for that state. A 
preliminary interview status was assigned after the clerical edit and before the computer edit. 
Data entry personnel were instructed to correct all errors identified during keying and to refer 
problem cases to their supervisor. One hundred percent independent verification was done. In 
Round 3, the pre-edit was incorporated into the clerical edit. 

6.7 MEASUREMENT ERROR 

Reinterviews were conducted in all rounds to obtain some indication of measurement error for 
the TFS. Reinterview sample sizes of current and former teachers, and overall response rates are 
presented in table 6-2. 



Table 6-2. — Reinterview sample sizes and response rates for current and former teachers: 
Rounds 1-3 



Category 


Round 1 
(1988-89) 


Round 2 
(1991-92) 


Round 3 
(1994-95) 


Total 


1,500 


1,498 


1,545 


Current teachers 


750 


708 


1,545 


Former teachers 


750 


790 


0 


Response rate (percent) 


81 


92 


63 



SOURCES: Jenkins and Wetzel (1995); Salvucci et al. (1997); Bushery, Schreiner, and Newman-Smith (1998). 



Overall the reinterview sample size was around 1,500 at each round. In the first two rounds, the 
sample size was divided fairly evenly between current and former teachers, but in Round 3 only 
current teachers were reinterviewed because of the desire to focus on the new instructional 
practices questions asked for current teachers. Additionally, only current teachers who were not 
reinterviewed for the Round 3 Teacher Survey were selected for the TFS reinterview. This 
practice was not followed in the previous two rounds. The reinterview sample was selected at 
the same time as the TFS sample in all rounds (Bushery et al., 1998). 

In Rounds 1 and 2, all reinterviews were conducted by telephone. When the TFS data were 
collected by phone interview, the interviewer who conducted the original interview also 
conducted the reinterview. In Round 1, for teachers who responded by mail prior to the cutoff 
date, the reinterviews were conducted by telephone from the U.S. Census Bureau’s Hagerstown, 
Maryland, Telephone Center. For all other teachers, U.S. Census Bureau field representatives 
conducted the reinterviews by telephone from regional offices (Jabine, 1994). In Round 2, the 
reinterviews were conducted about 2 to 3 weeks after the original interviews by supervisory field 
representatives. For Round 3, the mode of collection for the reinterview replicated the mode of 
collection for the original interview (Bushery et al., 1998). The reinterviews for mail 
respondents were conducted 3 to 4 weeks after the completion of the original response; if no 
response was obtained within 3 weeks of the reinterview mailing, a second reinterview 
questionnaire was mailed. There was no telephone followup of mail reinterview nonrespondents 
because it was felt that mode made a difference. Telephone reinterviews were conducted on a 
flow basis using a paper- and pencil-questiomiaire. U.S. Census Bureau field representatives 
completed the original TFS questionnaires and mailed them to regional offices; the regional 
offices then prepared and mailed reinterview questionnaires to senior U.S. Census Bureau local 
field representatives to conduct the telephone reinterviews. 

The response rate for the Round 3 reinterview was low at 63 percent, despite the good response 
rate of 71 percent by mail; the response rate for telephone cases was only 54 percent. To 
increase the telephone response rate, Bushery et al. (1998) have proposed that in the future the 
telephone reinterviews be carried out from telephone centers rather than by local field 
representatives. 
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In each of the first two rounds, two reinterview questionnaires were used — one for current 
teachers and one for former teachers. Although the Round 3 TFS reinterview program was 
confined to current teachers, it also used two different reiriterview questionnaires — one for mail 
respondents and one for telephone respondents (Whitener et al., 1999). In all three rounds the 
reinterview questionnaires contained a subset of questions from the original questionnaire that 
NCES deemed critical, that were suspected to be problematic, or were new questions not 
previously evaluated. The Round 1 current teacher reinterview questionnaire contained 32 items, 
mostly addressing teachers’ opinions, attitudes, and expectations, and the former teacher 
reinterview questionnaire contained 24 items (Salvucci et al., 1997). The Round 2 reinterview 
questionnaire contained 42 items for current teachers covering subject areas such as employment 
and teaching status, incentives and compensation, and background information; the former 
teacher reinterview questionnaire for the same round contained 21 items including questions on 
employment status, educational activities and future plans, and background information (Jenkins 
and Wetzel, 1995). In Round 3, the reinterview included many questions from the new teaching 
methods section (Bushery et al., 1998). 

The objectives of the reinterview program differed across survey rounds. For Rounds 1 and 3, 
the objective was simply to measure the degree of variability between the original responses and 
the reinterview responses (Whitener et al., 1999). The procedures used in the Round 2 
reinterview program were more extensive than those used in the other two rounds, in that they 
included a reconciliation component that aimed to determine the reasons for differences between 
the original and reinterview responses (Jenkins and Wetzel, 1995). 

In Round 2, all reinterview questions were asked before any reconciliation questions. After 
completing the reinterview questionnaire, interviewers were instructed to go back to the 
beginning of the questionnaire and to compare the respondent’s reinterview responses recorded 
on the left hand pages with the original responses, which had been recorded on the right hand 
pages of the reinterview questionnaire. When differences were detected between the original 
response and the reinterview response, reconciliation was attempted through a series of closed- 
ended probes tailored to each reinterview question (printed on the questionnaire) that aimed to 
identify the reasons for the differences (Harris, 1992a,b). An open-ended probe was used if a 
respondent did not choose one of the closed-ended probes. The open-ended responses were 
reviewed and clerically coded prior to data entry. The responses to the TFS were clearly visible 
to the interviewers as they conducted the reinterviews, and thus may have influenced the 
responses recorded to the reinterviews. This type of reinterview is known as a dependent 
interview (Jenkins and Wetzel, 1995). 

Round 1 reinterview results. When asked in reinterviews to report their status at the time they 
responded to the initial TFS interview, 83 (7 percent) of the teachers reinterviewed reported a 
status different than the one that they had reported in the initial interview. Of these 83 teachers, 
20 changed from current teacher in the initial interview to former teacher in the reinterview and 
63 changed from former to current teacher. No attempt was made to reconcile these differences 
in the reinterview. Because different sets of questions were asked for current and former 
teachers, those who reported a different status in the reinterview were excluded from further 
analyses of the questionnaire items included in the reinterviews (Royce, 1990). 
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Table 6-3 shows the distribution of estimated indexes of inconsistency for all items included in 
the TFS reinterviews in Round 1, separately for current and former teachers (see section 2.6.3 for 
a description of the index of inconsistency). Most of the reinterview items for former teachers 
dealt with the teachers’ opinions, attitudes and expectations. For current teachers there was a 
more nearly equal division between factual and opinion items. 



Table 6-3. — ^TFS reinterviews, by type and index of inconsistency: Round 1 



Index of inconsistency' 


Current 

(movers and stayers) 


Former 

(leavers) 


Factual 


Opinion 


Factual 


Opinion 


Total questions 


19 


13 


2 


22 


Low (0-19%) 


3 


0 


1 


0 


Moderate (20-50%) 


4 


1 


1 


7 


High (51-100%) 


2 


8 


0 


13 


Not available^ 


10 


4 


0 


2 



‘For items with more than two response categories, the L-fold index of inconsistency was estimated. 
^Did not meet the minimum requirements to compute a reliable estimate of the index of inconsistency. 
SOURCE: Royce(1990). 



Most of the factual items had indexes in the low or moderate ranges. The two factual items for 
current teachers that had high indexes of inconsistency related to teacher certification in the 
fields of their primary and secondary teaching assignments. The majority of opinion items had 
indexes of inconsistency in the high range and none of them were in the low range. 

Former teachers were asked to rate their current occupations on several aspects of job 
satisfaction both in an absolute sense and relative to teaching. Table 6-4 compares the indexes of 
inconsistency estimated for the absolute and relative ratings. Even though the indexes were in 
the moderate to high range for all items, respondents were clearly more consistent in providing 
comparative ratings on a three-point scale than they were in providing absolute ratings on a four- 
point scale. 

For items on current teachers' satisfaction with their jobs and on former teachers' satisfaction 
with their current jobs, all of which used a four-point scale, indexes of inconsistency were re- 
estimated with the four response categories collapsed into two: satisfied and dissatisfied. The 
resulting indexes were lower in all instances and in many cases moved from the high to the 
moderate range. As a result of these findings, the data from these items have generally been 
presented in the collapsed form in publications. 

Special analyses were conducted for the components of income reported by current teachers. 
These analyses showed, for those who reported non-zero amounts on both occasions, a 
correlation of 0.95 for reports of base salary. For other components the estimated correlations 
were much lower: 0.22 for non-teaching compensation and -0.39 for summer school salary. 



Table 6-4. — TFS index of inconsistency for selected opinion items, for former teachers: Round 1 





Index of inconsistency when: 


Aspect of current 
occupation rated 


Rated for 
current 
occupation' 


Current occupation 
compared to 
teaching^ 


Salary 


Point estimate 


363 


337 


90% confidence interval 


54-74 


30-48 


Opportunities for 


professional advancement 
Point estimate 


63 


56 


90% confidence interval 


54-75 


47-70 


Autonomy or control 


over your own work 
Point estimate 


’79 


’53 


90% confidence interval 


69-92 


43-65 


Benefits 


Point estimate 


’65 


’38 


90% confidence interval 


56-76 


31-48 


Intellectual challenge 


Point estimate 


60 


’43 


90% confidence interval 


51-72 


35-53 



‘Question 27: How satisfied are you with EACH of the following aspects of your CURRENT job? Are you (a) Very satisfied, (b) Somewhat 
satisfied, (c) Somewhat dissatisfied, or (d) Very dissatisfied with- 

^Question 26 : How would you rate teaching relative to your current PRIMARY occupation in terms of EACH of the following aspects? Please 
indicate (a) Better in teaching, (b) Better in current position, or (c) No difference — 

3 

Statistically significant difference between absolute and comparative ratings (at a 10 percent significance level). 

SOURCE: Royce(1990). 

Round 2 reinterview results: The Round 2 reihterview program for the TFS involved 

reinterviews followed by a reconciliation of any differences in responses. The reinterview data 
can be analyzed for consistency of response with the original data using the index of consistency 
as in Round 1 . Indexes of inconsistency were computed for 35 of the 42 items on the current 
teacher reinterview questionnaire (the other 7 items had too few cases to produce a reliable 
index) and for 15 of the 21 items on the former teacher reinterview questionnaire (Jenkins and 
Wetzel, 1995). All the items on both questionnaires were factual items. Two items on each 
questionnaire had indexes in the moderate range; all the others were in the low range. The two 
items in the moderate range on the current teacher questionnaire asked teachers to categorize 
their teaching assignment as regular, itinerant, or long-term substitute, and whether or not the 
teacher had a teaching certificate in their other teaching assignment field. The two items on the 
former teacher questionnaire asked about total combined income (in income categories) and 
persons other than spouse or children who depended on the respondent for more than half of their 
financial support. 
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The indexes of inconsistency in Round 2 were much lower than those in the previous round. 
Moreover, the consistency of response was significantly greater in Round 2 than in Round 1 for 
all but two of the 14 items that the two reinterview programs had in common. The before- 
reconciliation gross difference rates (the percent of respondents with different responses in the 
original interview and the reinterview) were significantly lower in Round 2 then in Round 1 for 
12 of the items (Jenkins and Wetzel, 1995). 

Jenkins and Wetzel (1995) suggest several possible reasons to explain the greater consistency in 
Round 2. A probable cause is the reconciliation component of the Round 2 reinterview program, 
with dependent interviewing. In some cases, the interviewers may have been influenced by the 
original responses in conducting the reinterviews, thus creating greater consistency. Inadequate 
interviewer training may have contributed to this behavior. 

As a result of the higher level of consistency of responses in Round 2, the reconciliation 
component of the reinterview program was able to identify only a limited number of reasons for 
observed differences. These reasons included misunderstanding the question, manual/general 
errors, category problems, and misunderstanding of the reference period. Given the limited 
success of the reconciliation component and its expense, it was not retained for Round 3. 

Results from the Round 2 reinterview and reconciliation process led to recommendations to 
rearrange and reword the answer categories for a question on the current teacher questionnaire 
concerning the type of teacher (regular, part-time, itinerant, etc.), and to rearrange the order of 
some questions on that questionnaire (Jenkins and Wetzel, 1995). However, these 
recommendations were not adopted in the following round because of insufficient time to test the 
changes before implementation. 

Round 3 reinterview results. The Round 3 reinterview program was confined to current teachers. 
Further, since the reinterview sample was not designed to measure mode effects, response 
variance measures were not computed separately for telephone and mail cases. Indexes of 
inconsistency were calculated for 65 questions; of these 51 (78 percent) displayed high levels of 
inconsistency, and only 3 (5 percent) displayed low levels (Bushery et al., 1998); Two questions 
were especially problematic. One asked whether the teacher's time was equally divided between 
two of several listed activities (e.g., teaching in an elementary or secondary school, working in 
such schools with an assignment other than teaching, working in an occupation outside of 
elementary or secondary school). This question had an index of inconsistency of 67. The second 
question, asking about the teacher's main activity during the work week, had an index of 
inconsistency of 62. These questions had been revised from those used in the 1991-92 TFS, but 
the new format remained problematic. 

The 1994-95 reinterview program again confirmed that "mark all that apply" questions are 
problematic. There were eight parts to a question that asked the teacher to describe 
characteristics (e.g., homogeneous, heterogeneous, remedial, gifted, special education, etc.) of 
his or her designated class; six had high response variance, and two had moderate response 
variance. Unclearly defined response categories may have exacerbated the usual problems of 
"mark all that apply" questions. 
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In the main activity section, the questions regarding teacher assignments and certificates showed 
the same problems as previously found in the Round 3 Teacher Survey reinterviews (Bushery, 
Schreiner, and Sebron, 1998). Further, all but one of the 54 questions on teaching methods, 
many of which were liew, had moderate or high response variance. 

The Round 2 and Round 3 reinterview programs for current teachers had five questions in 
common. For all of these, the index of inconsistency was lower in Round 2 than in Round 3. 
This finding is similar to that found in comparing the consistency of response between Rounds 1 
and 2. The explanation is probably a lack of independence of the initial and reinterview 
responses caused by dependent interviewing in the Round 2 reinterview program. 

Several recommendations for changes to the questionnaires were made based on the Round 3 
reinterview results (Bushery et al., 1998) including the following: 

■ Reconsider the objectives for the main activity questions and redesign the questions and test 
them. Cognitive researchers should participate in this work. At a minimum, "work week" 
should be defined more precisely to help focus respondents on the objective. 

■ Use "yes/no" response formats rather than the "mark all that apply" format. 

■ Test and reinterview revised questions to determine if reliability has improved. 

6.8 NONRESPONSE 

Unit nonresponse. Unit nonresponse to the TFS can arise at three stages: in obtaining teacher 
lists from schools; in obtaining response in the Teacher Survey from the selected sample of listed 
teachers; and in obtaining response in the TFS from a sample of those who responded in the 
Teacher Survey. 

Initially we consider the unit response rate associated with the last of these stages, that is, the 
percent of teachers selected from the Teacher Survey respondents who responded to the TFS. 
This response rate, which is termed the conditional TFS response rate, is displayed in table 6-5 
for all rounds, separately by sector and by status (current or former teacher). 

As the table shows, responses were obtained for the TFS in Rounds 1 and 2 from more than 90 
percent of the teachers selected in all categories. Round 3 had lower response rates than the 
previous two rounds in all categories where comparisons can be made. Because of difficulties in 
locating former teachers (leavers), their response rates are lower than those for current teachers 
(movers and stayers) (Kaufman, 1991). Response rates for public and private school teachers are 
not very different. 

The overall response rates for the TFS are the product of the conditional response rates in table 
6-5 and the overall response rates for the Teacher Survey. The overall rates for the TFS are 
presented in table 6-6. For teachers sampled from public schools, they range around 80 percent 
for the three rounds. They are lower for teachers sampled from private schools because of the 
appreciably lower response rates achieved for the Teacher Survey in private schools. In the first 
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Table 6-5. — Weighted conditional TFS response rates, by sector and teacher status: Rounds 1-3 


Weighted response rate (percent)* 


Sector and 
teacher type 


Round 1 
(1988-89) 


Round 2 
(1991-92) 


Round 3 
(1994-95) 


Total 


,97 


97 


92 


Current teachers 


97 


97 


92 


Former teachers 


94 


93 


89 


Public 


97 


97 


92 


Current teachers 


98 


97 


93 


Former teachers 


94 


92 


89 


Private 


96 


96 


87 


Current teachers 


97 


96 


87 


Former teachers 


93 


94 


88 


BIA 


t , 


t 


99 


Current teachers 


t 


t 


99 


Former teachers 


t 


t 


89 


*Not applicable. 








The basic weight is used to reflect the probability of selection. 






SOURCES: Bobbitt et al. (1991); Whitener et al. (1994); Whitener et al. (1999). 






Table 6-6. — ^Weighted overall TFS response rates, by sector and teacher status: 


Rounds 1-3 


Weighted response rate (percent)* 


Sector and 


Round ,1 


Round 2 , 


Round 3 


teacher type 


(1988-89) . . 


(1991-92) ( 


:i994-95) 


Public 


83 


84 


77 


Current teachers 


84 


84 , 


80 


Former teachers 


79 


80 , 


75 


Private 


72 


71 


64 


Current teachers 


72 


72 


64 


Former teachers 


71 


70 


64 



* The basic weight is used to reflect the probability of selection. 
SOURCES: NCES (1992); Whitener et al. (1994, 1999). 



two rounds, the overall response rates for teachers sampled from private schools are around 70 
percent, but the rate falls to 64 percent in Round 3. 
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Item nonresponse. Table 6-7, shows unweighted item response rates, for current and former 
teachers, for all rounds of the TFS. The rates for the two groups of teachers and for the three 
rounds are not directly comparable because of some differences in questionnaire content. 

Table 6-7. — Percent of items with selected response rates, by teacher status: Rounds 1-3 



Teacher status 


Percent of items with response rates: 
> 90 percent <80 percent 


Minimum of item 
response rate 
(percent) 


Round 1 (1988-89) 
Current 


90 


5 


65 


Former 


61 


1 


27 


Round 2 (1991-92) 


Current 


87 


5 


57 


Former 


95 


1 


67 


Round 3 (1994-95) 


Current 


90 


4 


52 


Former 


93 


1 


78 



SOURCES: NCES (1992); Whitener et al. (1994, 1999). 



In Round 1, a low-response item, which was common to both questionnaire versions, asked 
whether there were any persons, other than spouse and children, dependent on the responding 
teacher for more than half of their financial support. The response rate for this item was 49 
percent for both current and former teachers. For current teachers, only 65 percent responded to 
an item that applied only to movers who had moved to a private school, asking for the religious 
affiliation of that school. All other items on both versions of the questionnaire had response 
rates of 70 percent or more. 

Item response rates in Round 1 for a series of items asking for the level of respondents' 
satisfaction with various aspects of their current job were substantially higher for current 
teachers (all at about 99 percent) than they were for former teachers (83 to 91 percent) (Choy, 
Medrich, Henke, and Bobbitt, 1992, p. 154). Some former teachers, of course, did not have jobs 
for which these items would have been relevant, and the questionnaire had a skip instruction 
designed to allow them to bypass the items. The lower item response rates for former teachers 
may have been associated with some confusion about whether to skip and which set of items to 
skip. 

Three items on the questionnaire for current teachers in Round 2 had response rates less than 80 
percent. All of them related to earnings from nonteaching jobs. For former teachers, there was 
only one item, asking for the kind of business or industry where the respondent worked, with a 
response rate below 80 percent. 

Only one item on the former teacher questionnaire in Round 3 had a response rate of less than 80 
percent. This was the question: "In how many years do you plan to retire?’’ Several items had 
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response rates of less than 80 percent for the current teacher questionnaire. They included topics 
that had items with low response rates in previous rounds such as summer earnings in a 
nonteaching job and persons dependent on the teacher for more than half their financial support 
(Whitener et al., 1999). Such results suggest the need for additional questionnaire research on 
these topics. 

6.9 ESTIMATION 

Imputation. There was no imputation of missing items for Round 1. Missing values were left 
blank and hence treated as zeros. As a result, estimates of totals are generally underestimated. 
However, the imputations were later done and included on the CD-ROM (NCES, 1998) 
containing data from all three rounds. 

There were two stages of imputation performed in Round 2: logical imputation followed by hot 
deck imputation. In addition, a proration imputation procedure was applied to the current teacher 
questionnaires to adjust components of earned income so that they added to the total earned 
income. Logical imputation involved assigning values for missing responses that could be 
deduced from the teacher's other responses to the TFS and to the Teacher Survey. Logical 
imputation was used for about 26 percent of the imputed values for current teachers and about 4 
percent for former teachers. Proration imputation was used for 13 percent of the imputed values 
for current teachers. 

The second stage of imputation assigned values for the remaining missing responses by hot deck 
imputation. For items that were also included on the Teacher Survey, the hot deck procedure 
took account of the Teacher Survey response in order to yield more valid estimates of change 
between the two surveys. Donor values were assigned within imputation classes formed in terms 
of such variables as full-time/part-time status, instructional level for teachers, and then they were 
adjusted based on the recipient's and donor's values for the same item in the Teacher Survey. 
When the item was not also in the Teacher Survey, the same procedure was used but without the 
adjustment. Flags were assigned to identify imputed values and to distinguish between those 
imputed by hot deck methods and others. 

The imputation procedures employed in Round 3 were the same as those used in Round 2, except 
that an additional stage of clerical imputation was included. Clerical imputations were used 
when no suitable donor record was found, when the computer imputation produced an out of 
range entry, or when there were very few cases where an item was unanswered. The imputation 
flags distinguished between values that were not imputed, those imputed by ratio adjustment, 
those imputed by using other data on the record or from the individual's values in the Teacher 
Survey data file, those imputed by hot deck methods, and those imputed by clerical imputation. 

Weighting. The overall weights for teachers in Round 1 of the TFS were the product of the 
following three components: 

■ Teacher Survey final weight: the weight assigned to the teacher in producing the estimates for 
that survey (for a full description, see section 5.7.3); 
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■ TFS basic weight: the inverse of the teacher's probability of selection, given that he or she 
had been selected for the Teacher Survey; 

■ Nonresponse adjustment factor: an adjustment for eligible sample teachers for whom 
questionnaires were not obtained in the TFS. Within each of the 24 strata used in selecting 
the sample for the TFS, the nonresponse adjustments were calculated separately for each of 12 
adjustment cells defined by sex, level of education (2 categories) and age (3 categories) 
(Waite, 1990). 

The weights provided in the public-use data tape from Round 1 of the TFS were slightly different 
from those used to produce tabulations published in Characteristics of Stayers, Movers, and 
Leavers: Results from the TFS, 1988-89 (Bobbitt et al., 1991). The resulting changes in the 
estimates were very small relative to their standard errors; most of the published percents were 
not affected (Faupel, Bobbitt, and Friedrichs, 1992, pp. 17-18). 

In all rounds, a ratio adjustment was done that forces the weighted sample counts from the TFS 
to conform to the weighted number of teachers from SASS. Round 2 also had a SASS teacher 
within-school noninterview adjustment which accounted for teachers who were eligible for 
interview but did not participate in the 1990-91 Teacher Survey. Round 3 also had another 
weighting factor that was used to adjust for the fact that preliminary SASS final weights were 
used in computing the TFS basic weight. This factor adjusts for any changes that might have 
occurred between the preliminary and final weighting calculations. 

Variance estimation. As with other SASS surveys, a balanced half-sample variance estimation 
procedure was used to estimate sampling errors for the TFS in Rounds 1 and 2 and a bootstrap 
procedure was used in Round 3. Since the TFS sample is a subsample of the Teacher Survey 
sample, at each round the same set of replicates was used for both surveys, with a weight 
adjustment to compensate for the subsampling (see section 5.7 on variance estimation for the 
Teacher Survey and Whitener, Gruber, Rohr, and Fondelier, 1998, on variance estimation for the 
TFS). 

Tables 6-8 and 6-9 provide examples of the results of variance estimates computed by the 
bootstrap procedure for Round 3 expressed in terms of coefficients of variation (CVs), where the 
CV is the standard error of the estimate divided by the estimate. Table 6-8 presents Round 3 
estimates of the percents of stayers (teachers who are teaching in the same school as they were in 
the base year), movers (teachers teaching in a different school), and leavers (those who have left 
the teaching profession) for different categories of public school, together with the CVs for those 
estimates. Table 6-9 presents the corresponding information for base year teachers in private 
schools. 

With around 86 percent of the TFS sample froin public schools staying at the same school across 
all the categories of school in table 6-8, the CVs for the estimated percents of stayers are small, 
with nearly all being less than 2 percent. With only small percents of movers and leavers, the 
CVs for these estimates are much larger, and many exceed 10 percent. The percents of stayers 
and movers are slightly lower for teachers at private schools than for those at public schools and 
the percent of leavers is much larger (around 12 percent). In general, the CVs for the estimates 
from the TFS private school sample are larger than those for the estimates from the public school 
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Table 6-8. — Coefficients of variation (CVs) for estimates of public school teachers classified as 
stayers, movers, and leavers, by selected school characteristics: Round 3 









Public school teachers 








Stayers 


Movers 


Leavers 


Base year's school 
characteristics 


Percent 


CV 

(percent) 


Percent 


CV 

(percent) 


Percent 


CV 

(percent) 


Total 


86.3 


0.6 


7.2 


4.5 


6.6 


5.2 


School level 


Elementary 


85.4 


0.9 


7.8 


6.4 


6.8 


7.8 


Secondary 


87.5 


0.8 


5.7 


8.1 


6.7 


7.9 


Combined 


87.8 


2.1 


7.0 


20.3 


5.2 


20.0 


Not reported* 
School size 


87.4 


1.8 


8.0 


18.3 


4.6 


15.4 


Less than 150 


78.6 


4.4 


10.3 


19.7 


11.1 


32.4 


150-299 


85.8 


1.4 


7.0 


12.0 


7.1 


12.8 


300-499 


86.2 


1.2 


7.3 


10.4 


6.5 


11.4 


500-749 


85.3 


1.2 


7.1 


10.4 


7.6 


8.8 


750 or more 


87.7 


1.0 


6.6 


9.7 


5.7 


10.0 


Not reported* 


87.4 


1.8 


8.0 


18.3 


4.6 


15.4 


Minority status 


Less than 5% 


87.1 


1.2 


4.9 


11.0 


8.0 


11.6 


5 to 19% 


87.3 


1.2 


6.7 


9.6 


6.0 


11.8 


20 to 49% 


86.6 


1.1 


7.2 


10.1 


6.2 


13.2 


50% or more 


83.6 


1.4 


9.6 


9.0 


6.8 


10.6 


Not reported* 


87.4 


1.8 


8.0 


18.3 


4.6 


15.4 


Community type 


Central city 


86.1 


1.3 


7.6 


10.3 


6.3 


10.2 


Urban fringe/large town 


86.3 


1.1 


7.2 


8.9 


6.5 


10.8 


Rural/small town 


86.4 


0.8 


6.8 


6.0 


6.8 


8.8 


Region 


Northeast 


88.7 


1.5 


5.1 


13.5 


6.2 


14.2 


Midwest 


85.3 


1.1 


6.5 


10.2 


8.2 


9.0 


South 


85.1 


0.9 


8.4 


6.5 


6.5 


7.6 


West 


87.3 


1.2 


7.9 


10.0 


4.8 


14.3 



* Not reported data in this table are due to total survey nonresponse from some schools in Round 1 or Round 2 of SASS. 
SOURCE: CVs are computed using estimates and standard errors from Whitener et al. (1997). 
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Table 6-9. — Coefficients of variation (CVs) for estimates of private school teachers classified as 
stayers, movers, and leavers, by selected school characteristics: Round 3 



Private school teachers 

Stayers Movers Leavers 

Base year's school CV CV CV 



characteristics 


Percent 


(percent) 


Percent 


(percent) 


Percent 


(percent) 


Total 


82.3 


1.0 


5.8 


6.0 


11.9 


5.9 


School level 


Elementary 


83.7 


1.3 


5.8 


8.6 


10.5 


9.1 


Secondary 


83.0 


2.1 


5.5 


15.1 


11.5 


13.1 


Combined 


79.1 


2.8 


6.0 


15.0 


14.9 


12.3 


Not reported* 


82.8 


3.7 


5.5 


23.0 


11.6 


24.1 


School size 


Less than 150 


76.0 


2.6 


7.9 


13.3 


16.2 


10.5 


150-299 


79.6 


2.4 


8.0 


14.1 


12.4 


12.0 


300-499 


84.3 


1.8 


3.2 


20.0 


12.5 


11.7 


500-749 


89.6 


1.7 


3.3 


24.3 


7.1 


17.9 


750 or more 


89.6 


2.0 


4.2 


29.0 


6.2 


19.0 


Not reported* 


82.8 


3.7 


5.5 


23.0 


11.6 


24.1 


Minority status 


Less than 5% 


83.9 


1.5 


5.1 


13.5 


11.0 


9.6 


5 to 19% 


84.8 


1.5 


4.2 


12.6 


11.1 


9.6 


20 to 49% 


77.0 


3.4 


7.4 


17.4 


15.6 


14.6 


50% or more 


75.8 


3.4 


10.9 


14.0 


13.2 


16.3 


Not reported* 


82.8 


3.7 


5.5 


23.0 


11.6 


24.1 


Community type 


Central city 


83.4 


1.1 


5.7 


8.6 


10.9 


6.9 


Urban fringe/large town 


81.9 


1.9 


5.5 


14.4 


12.6 


10.5 


Rural/small town 


80.0 


2.3 


6.4 


16.9 


13.6 


10.7 


Region 


Northeast 


84.0 


1.8 


7.1 


12.8 


8.9 


14.7 


Midwest 


84.3 


1.8 


5.9 


9.5 


9.8 


12.2 


South 


81.5 


2.0 


4.1 


11.7 


14.5 


10.0 


West 


78.3 


3.0 


6.7 


23.9 


15.0 


12.0 



* Not reported data in this table are due to total survey nonresponse from some schools in Round 1 or Round 2 of SASS. 
SOURCE: CVs are computed using estimates and standard errors from Whitener et al. (1997). 




174 



191 



sample. Nevertheless, the CVs for the estimates of stayers in private schools are all less than 4 
percent. However, the CVs for the estimates of movers and stayers from many categories of 
private school are large. 
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7. THE LIBRARY SURVEY 



7.1 INTRODUCTION 

The 1993-94 Round (Round 3) of the Schools and Staffing Surveys (SASS) added two new 
surveys that expanded the survey’s analytic potential to address policy needs. One of these 
surveys is the Student Records Survey that collects data for a sample of students in SASS 
schools. That survey is described in chapter 8. The other survey is the Library Survey, which is, 
in effect, two surveys, with separate data collections for school library media centers and school 
library media specialists/librarians (generally abbreviated subsequently to libraries and 
librarians). Since these two surveys use the same methodology and are conducted in the same 
sample of schools, they are described together in this chapter. They are, henceforth, referred to 
as the Library Survey and the Librarian Survey. The Library Survey has been repeated in Round 
4, but the Librarian Survey has not. 

Over the past 10 years, school libraries have become a topic of interest and concern to education 
policymakers and researchers (Ingersoll, 1995). Round 2 of SASS obtained limited basic 
information about the availability, staffing, and role of libraries in the School Survey. However, 
it could not address issues such as the background, training, and attitudes of school librarians, 
and the expenditures, materials, equipment, and services provided by school libraries. Round 3 
of SASS followed this initial effort by including the Library and Librarian Surveys to collect 
information on these topics. Data from these surveys provide the most comprehensive and 
extensive database on school libraries since the NCES 1985 Survey of Public and Private School 
Libraries and Media Centers. 

The purpose of the Library Survey was to provide a national picture of school library media 
centers and to assess the adequacy of these centers in meeting the needs of students and staff. 
The goal of the Librarian Survey was to obtain data that would describe school librarians in 
terms of their educational background, work experience, and demographic characteristics, as 
well as their duties, salary, workload, and attitudes about their current position and their 
profession. Comparisons between librarians and teachers can be made since many questions 
asked in the Librarian Survey and the Teacher Survey were similar. Both surveys covered 
public, private, and Bureau of Indian Affairs (BIA) schools and were designed to produce 
estimates at the state level for public schools and at the major affiliation level (Catholic, other 
religious, nonsectarian) for private schools. 

This chapter is organized in the same manner as the previous chapters — that is, with a section for 
each major survey activity: sampling frames (7.2), sample design (7.3), content (7.4), data 
collection procedures (7.5), data processing (7.6), measurement error (7.7), nonresponse (7.8), 
and estimation (7.9), and evaluation of estimates (7.10). 

7.2 SAMPLING FRAMES 

The target population for the Library Survey consisted of the library media centers in all the 
schools eligible for the School Survey (see section 2.1), except that library media centers in 
private schools with special program emphasis, or with special education, vocational or 



alternative curricula were excluded, because it was felt that the libraries in these schools would 
be quite different from those in regular schools. For the purposes of the Library Survey, a library 
media center was defined as: 

“an organized collection of printed and/or audiovisual and/or computer 
resources which (a) is administered as a unit, (b) is located in a 
designated place or places, (c) makes resources and services available to 
students, teachers, and administrators. It is this definition, not the name, 
that is important; it could be called a library, media center, resource 
center, information center, instructional materials center, learning 
resource, center, or some other name.” (Gruber, Rohr, and Fondelier, 

1996, 1993-94 SASS Data File User’s Manual, page F-213). 

In the rest of the chapter, the term “library” will be used to mean “library media center.” 

The target population for the Librarian Survey comprised the school employees who were 
responsible for the schools’ library media centers, that is, the library media specialists or 
librarians. Henceforth, such persons will be termed the school librarians. 

7.3 SAMPLE DESIGN 

The Library and Librarian Surveys were conducted in the same sample of schools. This sample 
was selected from the schools sampled for the School Survey. The sample comprised a 
subsample of public and private schools (excluding private schools with special program 
emphasis or with special education, vocational or alternative curricula), and all BIA schools in 
the School Survey. The overall sample size was 7,670 schools. The school sample sizes by 
sector and school level are displayed in table 7-1. These school sample sizes exceed the numbers 
of libraries sampled because not all sampled schools had libraries. Both for this reason and 
because not all school libraries had librarians, the school sample sizes also exceed the number of 
librarians sampled. Of the schools sampled for the Library and Librarian Surveys, 371 public 
schools, 465 private schools, and 18 BIA schools were out of scope or did not have libraries; 851 
public schools, 1,178 private schools, and 49 BIA schools did not have librarians either because 
they were out of scope, because they did not have a library, because their library did not have a 
librarian, or because the librarian’s main assignment at the school was some other position. 

Table 7-1. — Sample schools selected for the Library and Librarian Surveys, by type of school; 

Round 3 



School type 


Elementary 


Combined 


Secondary 


Total 


Total 


3,773 


1,400 


2,537 


7,670 


BIA 


122 


29 


25 


176 


Non-BIA 

Public 


2,274 


698 


2,022 


4,994 


Private 


1,337 


673 


490 


2,500 



SOURCE: Gruber et al. (1996). 
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The basic sample design for selecting the schools for the Library and Librarian Surveys was the 
same for public and private schools. In each case, it was a stratified systematic sample from an 
ordered list with schools being selected with unequal probabilities. Public schools were stratified 
by state and grade level, and ordered within strata by local education agency (LEA) metro status, 
1991-92 LEA Common Core Data Identification (CCD ID), school enrollment, and the 1991-92 
School CCD ID. Private schools were stratified by recoded affiliation (Catholic, other religious, 
nonsectarian), grade level, and urbanicity and ordered within strata by frame (list or area) and 
school enrollment. For both public and private schools, the measure of size used for the unequal 
selection probability was the school’s measure of size used for selection into the School Survey 
divided by the school’s selection probability for that survey. The use of this measure of size for 
subsampling schools tends to cancel out the differential selection probabilities across the strata in 
the School Survey. Any school with a measure of size larger than the sampling interval used in 
its stratum was selected with certainty. 

7.4 CONTENT 

Libraries. The Library Survey questionnaire had five sections or content areas: 

Q Library media center staffing: counts of certified library media specialists, professional staff 
members who were not certified as library media specialists, and other paid library staff by 
full-time/part-time status; college degrees held by professional staff members; number of 
adult and student volunteers in the library; and, for private schools, counts of staff members 
who worked on a contributed basis. 

□ 1992-93 collection and expenditures: books, subscriptions, audio-visual materials, computer 
software, and CD-ROM titles, acquired during, and held at the end of, the school year; 
expenditures by type; and adequacy of the collection for the school’s needs. 

Q Technology: technical equipment and services available in the school (library or elsewhere), 
(e.g., computer with modem, CD-ROMs, Internet connections, cable television, prerecorded 
videotapes). 

Q Library media center facilities: library seating capacity and other types of space in the 
library, such as conference rooms and workrooms. 

□ Scheduling and transactions: classes scheduled in the library media center; students” use of 
the library media center; and the extent to which materials can be checked out by students. 

Librarians. The Librarian Survey questionnaire covered the following topics: 

H Current status, including whether the librarian was full time or part time, whether the 
librarian was also a classroom teacher, other assignments in the school, and the librarian’s 
main activity outside the school. 

a Experience and training. 

E Collaborative activities, such as working with classroom teachers, and other duties at school. 



0 Perceptions and attitudes toward work. 

H Salary, other earned income, benefits, and family income. 

a Demographic information, gender, race/ethnicity, year of birth, marital status, and 
dependents. 

There were different library questionnaires and also different librarian questionnaires for public 
schools, private schools, and BIA schools. 

7.5 DATA COLLECTION PROCEDURES 

Field test. A field test of the Library and Librarian Surveys was conducted during Round 2 of 
SASS. The library and librarian questionnaires were mailed to a subset of the SASS sample 
schools. Nonresponse followup was conducted by U.S. Census Bureau field representatives who 
conducted the interviews by telephone. Almost 300 questionnaires were mailed to libraries and 
to librarians in public schools, and almost 400 were mailed to libraries and to librarians in private 
schools. Over 95 percent of eligible libraries and librarians responded from public schools. The 
response rates for private schools were lower, at about 89 percent for eligible libraries and 86 
percent for eligible librarians. A notable finding was that 53 percent of sampled private schools 
did not have an eligible librarian. This percent includes cases where the school was out of scope, 
the school did not have a library, and the library did not have a librarian. Some items with low 
response rates in the field test were dropped from the main survey questionnaires, while others 
were reworded as a result of the field test results (Williams, 1992; Gruber et al., 1996). 

Procedures. The library and librarian questionnaires were mailed to sampled schools in October 
1993. The library questionnaire was addressed to the “Principal,” but the respondent could be 
the librarian or another staff member familiar with the library. The librarian questionnaire was 
addressed to the “Library Media Specialist/Librarian” and was to be completed by the school 
employee who was responsible for the school’s library. As for all other SASS surveys in Round 
3, reminder postcards were mailed out about 1 week after the initial mailing. 

The second mailing of questionnaires was sent out in November/Decembef to those who had not 
returned the completed questionnaires by that time. In this mailing, library questionnaires were 
sent to 56 percent of public schools sampled, 67 percent of private schools sampled, and 59 
percent of BIA schools sampled. Slightly lower percents of librarian questionnaires had to be 
resent: to 51 percent of public school librarians, to 64 percent of private school librarians, and to 
56 percent of BIA school librarians. 

Computer-assisted telephone interviewing (CATI) followup of mail questionnaire 
nonrespondents began about 6 weeks after the second mailing of questionnaires. It was 
conducted between January and June 1994 from two centralized locations. Table 7-2 shows the 
numbers and percents of libraries and librarians requiring CATI followup and the number of 
successes, as well as the cases that could not be completed with CATI. As shown, about one- 
third of libraries and librarians had to be followed up by CATI, and success was generally good, 
except in the case of private school librarians, who had a much lower percent of CATI 
completions than the other groups. Some cases that could not be completed in CATI were 
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interviewed by telephone by local U.S. Census Bureau field representatives. These field 
representatives also had generally high success rates, except again with private school librarians. 

Table 7-2. — ^Libraries and librarians selected for and completing telephone interview followup, 
by followup interview and type of school: Round 3 



CATI followup Non-CATI followup 





Total 


Completed 




Total 




Completed 




School type 


Number 


Percent' 


Number Percent^ 


Number Percent^ 


Number Percent'' 


Libraries 


















Public 


1,940 


39 


1,143 


59 


385 


8 


365 


95 


Private 


1,098 


43 


604 


55 


159 


6 


125 


79 


BIA 


57 


36 


43 


75 


9 


6 


8 


89 


Librarians 


















Public 


1,382 


28 


990 


72 


342 


7 


274 


80 


Private 


1,004 


40 


354 


35 


186 


7 


80 


43 


BIA 


51 


32 


29 


57 


8 


5 


7 


88 



‘Percent of sample cases in CATI followup. 

^Percent of sample cases in CATI followup that were completed. Cases identified as noninterview and out of scope for the survey during the 
CATI followup are not included. 

^Percent of sample cases in telephone followup by field representatives. 

^Percent of sample cases in telephone followup by field representatives that were completed. Cases identified as noninterviews and out of scope 
for the survey during followup are not included. 

NOTE: Numbers do not total because of noninterview out-of-scope cases and because nonresponse cases may be switched between CATI and 
non-CATI followup. 

SOURCE: Gruber et al. (1996). 



7.6 DATA PROCESSING 

The Library and Librarian Surveys underwent the usual SASS data processing, consisting of the 
assignment of an initial interview status code, a clerical edit, being grouped for keying, 100 
percent verification with all errors corrected, and a computer edit, before the assignment of the 
final interview status recode values. Computer pre-edits were not performed for either survey. 

Libraries. For all three types of library questionnaire (public, private, and BIA), more than 50 
percent of the variables (items requiring a response) that were edited had change rates of at least 
10 percent. For example, an item with 433 or more changes (corrections or deletions) among the 
4,321 public library records computer edited would have a change rate of 10 percent or more. 
Additionally, for all three types of questionnaires, at least 10 percent of the variables had change 
rates of 50 percent or more, implying that many respondents had problems with these variables. 
Three questions with the same format asked for the numbers of state-certified library media 
specialists, noncertified professional staff members, and other paid workers according to the 
fraction of time they worked in the library (categorized in quarters, e.g., at least one-half time but 
less than three fourths time). The responses to all categories of these questions, except the 
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“None” response, were subject to very high rates of change. Fondelier (1996) recommended that 
the question wording needed to include the word “library” when asking about the types of staff 
working in the library, as teachers were often reported. Another example of a question with a 
very high change rate was a “mark all that apply” question that asked about the types of available 
space in the library. All but two of the 10 categories listed in this question had high change 
rates; the two that did not were. the “None” category and the “Individual reading, viewing, and 
listening” category. 

Librarians. The rate of edit changes was smaller for the librarian questionnaires than for the 
library questionnaires. For all three types of school librarians, about 30 percent of the edited 
variables had change rates of 10 percent or more, and 5 percent or less had change rates greater 
than 50 percent. One “mark all that apply” question asked the librarian about the school levels — 
prekindergarten, elementary (including kindergarten), middle school or junior high, senior high, 
and postsecondary — at which he or she had worked for a year or more. The responses to the 
three middle levels had high change rates across all three sectors. 

7.7 MEASUREMENT ERROR 

Some reinterviews were conducted for a subset of questions from the library questionnaires, but 
none were done for the librarian questionnaires. The data collection method for the library 
reinterviews was matched to that used in the original data collection; the reinterviews were by 
mail questionnaire for schools that responded to the original questionnaire by mail and by CATI 
interview from a centralized location for those that were originally interviewed by CATI. For 
the telephone reinterviews, the interviewers attempted to reinterview the respondent who 
completed the original questionnaire. 

A systematic subsample of 1,780 schools (833 public schools and 947 private schools) was 
-selected from the original sample for reinterview. Reinterview samples were selected before the 
mailout of survey questionnaires, so the sample includes losses due to original survey 
noninterviews (Gruber et al., 1996). Out of the targeted number, only 1,343 were eligible (i.e., 
the school was not out of scope and had a library), and of these, 959 reinterviews were obtained 
for a response rate of about 72 percent. The subset of questions in the reinterview questionnaire 
comprised 32 items, many of which had multiple response categories. These items collected 
information on library staffing, 1992-93 collection and expenditures, technology, library media 
center facilities, and scheduling and transactions. 

Salvucci et al., (1997) provides a summary of the library reinterview results by the five sections 
of the questionnaire described above. The results are expressed in terms of the L-fold index of 
inconsistency; values of the index less than 20 may be considered as low, values between 20 and 
50 as moderate, and values over 50 as high (see section 2.6.3 for a description of the index). 
None of the five sections had low average values of the index for the items included in the 
reinterviews. The Technology Section had the lowest average value for the index at 23, followed 
by the Scheduling and Transactions Section with an average of 32, and the Collection and 
Expenditures Section with an average of 37. The Staffing Section had an average moderate 
value, and the Facilities Section had a high average value of 52. 
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The analyses of the reinterviews identified a number of problematic questions and led to 
suggestions for changes in the future. For example: 

□ Three questions asked for the number of professional staff of different types working in the 
library media center in categories according to the fraction of time spent (e.g., full time, at 
least three-fourths time but less than full time). These are the same items that had high edit 
change rates discussed above. Respondents had difficulty with the time categories and two 
of the items had high indexes of inconsistency (59.3 and 50.7). Improved definitions of the 
categories were recommended. In Round 4, the “Vi to V 2 time” and the “less than Vi time” 
categories were dropped as these were found to be the most problematic by the reinterviews. 

a The responses to the questions on the numbers of materials of different, types (e.g., books, 
video materials) the library acquired in the past year and held at the end of the year were 
observed to heap at rounded estimates. They were also subject to moderate levels of 
inconsistency. It appeared that respondents had difficulty giving consistent and exact 
numbers. Recommendations were made to simplify this task. The matrix that asks for these 
library holdings was revised for Round 4. 

B Some definitional changes were recommended in the Technology Section. The number of 
microcomputers is not asked for in Round 4. 

B The question on types of space available in the library media center was in a “mark all that 
apply” format. As noted above, this question was also subject to high edit change rates. The 
reinterviews showed that all the responses had moderate or high levels of inconsistency. The 
seven items with a high level of inconsistency had index values that ranged from 54.9 to 
68.2. A recommendation was made to avoid the “mark all that apply” format in the future. 
A “Yes/No” response format was used for each type on space available in Round 4. 

7.8 NONRESPONSE 

Unit nonresponse. Weighted unit response rates for both the Library and Librarian Surveys are 
presented by state for public schools in table 7-3 and by affiliation group for private schools in 
table 7-4. Throughout the rates are computed for in-scope schools, excluding schools without 
libraries and special education libraries in private schools for the Library Survey, and also 
libraries without librarians for the Librarian Survey. 

The overall \yeighted response rate for the Library Survey was about 86 percent, with the rate of 
90 percent for public schools being much higher than the rate of 71 percent for private schools. 
Fondelier (1996) suggested that the relatively low response rate for private schools reflected 
response burden, in that the same schools were selected to respond to the School Survey and 
were asked to provide a substantial amount of detailed information. Further, some small schools 
with small libraries may have believed that the survey did not apply to them because of the many 
technology questions. This lower response rate for libraries of private schools should be noted 
when doing analysis. By state, the rates for public school libraries varied between 80 percent for 
Nebraska and Louisiana and 98 percent for Alabama and Florida, except for Alaska that had a 
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Table 7-3. — Weighted response rates for the public school Library Survey and the Librarian 
Survey, by state: Round 3 



Weighted response rate (percent)* Weighted response rate (percent)* 



State 


Library Survey 


Librarian Survey 


State 


Library Survey 


Librarian Survey 


Total 


90 


92 








Alabama 


98 


96 


Montana 


86 


90 


Alaska 


74 


81 


Nebraska 


80 


85 


Arizona 


97 


95 


Nevada 


90 


90 


Arkansas 


97 


92 


New Hampshire 


96 


' 98 


California 


82 


88 


New Jersey 


83 


95 


Colorado 


89 


87 


New Mexico 


90 


94 


Connecticut 


87 


96 


New York 


85 


92 


Delaware 


90 


94 


North Carolina 


91 


96 


District of Columbia 


85 


86 


North Dakota 


83 


88 


Florida 


98 


94 


Ohio 


89 


86 


Georgia 


96 


94 


Oklahoma 


90 


94 


Hawaii 


97 


96 


Oregon 


95 


94 


Idaho 


95 


95 


Pennsylvania 


92 


99 


Illinois 


89 


91 


Rhode Island 


92 


95 


Indiana 


97 


97 


South Carolina 


92 


98 


Iowa 


97 


96 


South Dakota 


81 


91 


Kansas 


95 


96 


Tennessee 


91 


86 


Kentucky 


; ^ ^ 85 


90 


Texas 


95 


87 


Louisiana 


80 


98 


Utah 


95 


96 


Maine 


90 


91 


Vermont 


92 


96 


• Maryland 


90 


95 


Virginia 


92 


91 


Massachusetts 


88 


89 


Washington 


96 


97 


Michigan 


91 


91 


West Virginia 


85 


94 


Minnesota 


92 


97 


Wisconsin 


92 


96 


Mississippi 


91 


91 


Wyoming 


87 


95 


Missouri 


93 


96 









•The basic weight is used to reflect the probability of selection. 
SOURCE: Gruber et al. (1996). 
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Table 7-4. — Weighted response rates for the private school Library Survey and the Librarian 
Surveys, by affiliation group: Round 3 



Weighted response rate (percent)* 


Affiliation group 


Library Survey 


Librarian Survey 


All private schools 


71 




77 


Catholic 


82 




86 


Parochial 


80 




85 


Diocesan 


86 




85 


Private Order 


84 




92 


Other Religious 


59 




58 


Conservative Christian 


58 




71 


Affiliated 


66 




66 


Unaffiliated 


53 




34 


Nonsectarian 


70 




88 


Regular 


75 




90 


Special emphasis 


56 




79 


Special education 


$ 




$ 



♦The basic weight is used to reflect the probability of sample selection. 

*Not applicable. (Data were not collected from special education libraries; those libraries were out of scope.) 
SOURCE: Gruber et al. (1996). 



rate of only 74 percent. Among private schools, the rate for Catholic school libraries (82 
percent) was much higher than that for other religious school libraries (59 percent) and 
nonsectarian school libraries (70 percent). 

The higher response rate for public schools than for private schools observed in the Library 
Survey is also found in the Librarian Survey: the weighted response rate of 92 percent for public 
school librarians is appreciably higher than the 77 percent for private school librarians. Again, 
this lower response rate for librarians in private schools should be noted. For public schools, the 
rate varies by state from a low of 81 percent in Alaska to almost complete response in 
Pennsylvania. For private schools, the response rate is only 58 percent for librarians in other 
religious schools, as compared with 86 percent for those in Catholic schools and 88 percent for 
those in nonsectarian schools. The response rates for BIA schools were 89 percent for the 
Library Survey and 88 percent for the Librarian Survey. 

As with other SASS surveys, Monaco et al. (1998) conducted analyses to examine variation in 
response rates in the Library and Librarian Surveys by school characteristics. Some results are 
presented for both surveys in table 7-5, including the significance levels reached using an overall 
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Table 7-5. — Weighted response rates for Library and Librarian Surveys, by selected 
characteristics: Rounds 





Library Survey 

Weighted response rate (percent)* 


Librarian Survey 

Weighted response rate (percent)* 


Characteristics 


Public 


Private 


Public 


Private 


Region: 










West 


88 


69 


91 


78 


South 


92 


71 


92 


77 


Northeast 


88 


63 


94 


68 


Midwest 


91 


78 


93 


83 


Urbanicity: 


P=0.058 


P<0.001 


P=0.263 


P=0.004 


Central city 


88 


74 


90 


81 


Urban firinge/large town 


88 


70 


92 


74 


Rural 


92 


66 


94 


72 




P=0.026 


P=0.053 


P=0.036 


P=0.061 


School level: 


Combined 


76 


61 


90 


67 


Secondary 


92 


86 


94 


91 


Elementary 


90 


72 


92 


76 




P<0.001 


P<0.001 


P=0.011 


P<0.001 


School size: 


1-149 


80 


52 


91 


44 


150-499 


90 


81 


92 


84 


500-749 


93 


87 


91 


93 


750+ 


92 


92 


95 


91 




P<0.001 


P<0.001 


P=0.172 


P<0.001 


School type: 


Nonregular 


53 




86 




Regular 


91 

P<0.001 




92 

P=0.103 




Minority enrollment: 


<5.5% 


91 




93 




5.5-20.5% 


92 




94 




20.5-50.5% 


90 




92 




>50.5% 


86 

P=0.019 




88 

P=0.008 




School sampled in 1990-91: 


Yes 


89 


81 


91 


84 


No 


91 


68 


93 


74 




P=0.351 


P<0.001 


P=0.463 


P=0.001 


1991-92 PSS status: 


Respondent 




72 




79 


Nonrespondent 




39 




32 


Not in 1991-92 PSS 




51 

P<0.001 




45 

P<0.001 



*The basic weight is used to reflect the probability of sample selection. 
*Not applicable. 

SOURCE: Monaco et al. (1998). 
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test (Rao-Scott chi-squared statistic) for differences in rates across the categories of the variable 
concerned. The results are generally comparable across surveys. The public and private schools 
in both the Library and Librarian Surveys show significant differences by category for school 
level, with highest response rates for secondary schools throughout. Except for public schools in 
the Librarian Survey, school size is significantly associated with survey response, with the 
smallest schools having lower response rates. There are significant regional differences in 
response rates for private but not public schools in both surveys, with lower rates in the 
Northeast. Minority enrollment and urbanicity are significantly associated with response for 
public schools in both surveys. The association of urbanicity and response is almost significant at 
the 5 percent level for private schools in both surveys, but the direction of the association is the 
opposite of that for public schools: rural schools had the highest rates for public schools but the 
lowest rates for private schools. The response rate for librarians in private schools that were 
nonrespondents in the 1991-92 Private School Survey is extremely low for both the Library 
Survey (39 percent) and the Librarian Survey (31 percent). 

Further analyses conducted by Monaco et al. (1998) showed significant associations between 
response status in the Library Survey and response status in the Librarian Survey for both public 
and private schools. Also the response status for both these surveys was significantly associated 
with the school’s response status in the School Survey and in the Principal Survey. For public 
schools, there was also a significant association of response status for both surveys with the 
response status of the school’s LEA in the Teacher Demand and Shortage Survey (TDSS). 

Item nonresponse. Table 7-6 provides some summary information about unweighted item 
response rates for the two surveys. Although there were few missing data for most items in both 
surveys, the summary information in the table indicates that there were some problematic items. 

Table 7-6. — Percent of items with selected response rates in the Library and Librarian Surveys, 
by sector: Round 3 





Percent of items with response rates: 


Minimum of item 
response rate 
(percent) 


Sector 


> 90 percent 


<75 percent 


Libraries 


Public 


81 


5 


57 


Private 


80 


4 


66 


BIA 


82 


1 


61 


Librarians 


Public 


87 


6 


61 


Private 


80 


11 


50 


BIA 


87 


5 


56 



SOURCE: Gruber et al. (1996). 
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The item in the Library Survey that had the lowest item response rate for all three sectors 
(around 60 percent) asked for the number of students using the library in the most recent full 
week of school. Fondelier (1996) suggested that this item should be deleted from future data 
collections because of the low response rate. Other items with low response rates were the 
elements of a grid in which the respondent was asked to record acquisitions, holdings, and 
expenditures for a set of materials (e.g., books, microcomputer software) for the past school year. 
The response rates for most of these items were less than 75 percent. As noted above, the 
reinterview program showed that when this information was provided, it had a moderate level of 
inconsistency. 

The set of items with the lowest response rates in the Librarian Survey were items asking the 
librarian to report the frequency (weekly, monthly, annually or never) with which they worked 
with the school’s classroom teachers to plan units of instruction in different subject areas. The 
later subjects in the list (arts and humanities, health/physical education, vocational/technical 
education, guidance, and English/language arts) had particularly low response rates, averaging 
around 63 percent for public school librarians, 54 percent for private school librarians, and 58 
percent for BIA school librarians. This question is asked in Round 4 with a different matrix 
format. 

Another item with a particularly low response rate appeared only on the questionnaire for private 
school librarians. It asked whether or not the librarian worked in the school on a contributed 
service basis for less than full salary or no salary; only 65 percent of librarians responded. Both 
for public and private school librarians, items asking about the major field and year earned for 
education specialist or professional diplomas and for doctorates had response rates of less than 
75 percent. 

7.9 ESTIMATION 

Imputation. The same three stages of imputation were used for the Library and Librarian 
Surveys. At the first stage, items were completed wherever possible by using information from 
other questionnaire items on the data record, from the matching library or librarian questionnaire 
or, for libraries, from the matching SASS school questionnaire. At the second stage, a hot deck 
imputation method was used to fill ih missing data by using data from a similar library or 
librarian record. For some items, the values were simply taken from the donor record but, in 
other cases, the donor’s value was used as a factor along with other information on the 
incomplete record to assign a value for a missing response. Different sorting variables were used 
for public, private and BIA school libraries and librarians for matching recipients and donors. In 
a few cases, when the number of missing values was small or when no suitable donor could be 
found, the missing values were completed by clerical imputation. Imputation flags were 
employed to denote the stage of imputation as for other SASS surveys. 

Weights. The weighting procedures for the Library and Librarian Surveys were of the same 
general form. In each survey, the weights were computed separately for public, private and BIA 
schools. The final weights are the product of the following factors: 

■ School basic weight: the inverse of the probability of selection for the School Survey. 
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■ Library subsampling factor: an adjustment that accounts for the subsampling of schools from 
the School Survey sample for the Library and Librarian Surveys. 

■ Sampling adjustment factor: an adjustment that accounts for unusual circumstances that 
affect the school’s probability of selection for the School Survey (such as splits, mergers, or 
duplication). 

■ Type A noninterview adjustment factor: an adjustment that accounts for library 
nonrespondents where it is not known whether or not the school had a library and where, for 
the Librarian Survey, it accounts for library nonrespondents that did not report whether or not 
they had a library and the librarian was a refusal or unable to contact. 

a Type B noninterview adjustment: an adjustment that accounts for library nonresponse (for 
the Library Survey) dr librarian nonresponse (for the Librarian Survey), when the school is 
known to have a libr^y. 

a Type C noninterview adjustment factor (for the Librarian Survey only): an adjustment that 
accounts for librarian nonresponse when librarian status is known. 

a First-stage ratio adjustment factor: a factor that adjusts sample estimates for the total number 
of noncertainty schools in the Library and Librarian Surveys to corresponding frame totals. 

a Second-stage ratio adjustment factor: In the case of the Library (Librarian) Survey this factor 
is applied using information about the presence of a library (librarian) only from the School 
Survey (information that may differ from that reported in the Library and Librarian Surveys). 
The adjustment is the ratio of the weighted count of schools with libraries (librarians) from 
the School Survey to the weighted count of schools with libraries (librarians) in the Library 
(Librarian) Survey sample. A corresponding adjustment is made for schools without libraries 
(librarians). 

The weighting adjustments outlined above were performed within adjustment cells, with 
collapsing of cells when the adjustments were too large or the cell sample sizes were too small. 
For example, for public schools, the Type A and Type B adjustment factors were applied within 
cells defined by state by grade level by enrollment by urbanicity, and cells were collapsed if the 
adjustment factor exceeded 1.5 or the number of schools was less than 10. Details of the various 
adjustment cells are provided by Gruber et al. (1996, pp. 296-300). 

Variance estimation. As with most other Round 3 SASS surveys, a bootstrap method of variance 
estimation is used for the Library and Librarian Surveys (see section 2.7.5). The bootstrap 
replicate weights are the corresponding weights in the School Survey sample adjusted for the 
subsampling. However in two situations the balanced half-sample replication is used in place of 
the bootstrap: these are cases of schools selected with certainty for the School Survey, but 
subsampled for the Library and Librarian Surveys, and private schools sampled from the area 
frame for the School Survey (Abramson et al., 1996). 

Tables 7-7 and 7-8 present estimates from the Library Survey of the percent of schools with 
libraries that have a computer and modem, the percent rating the current support in science and 
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Table 7-7. — ^Coefficients of variation (CVs) for selected items from the public school Library 
Survey, by selected school characteristics: Round 3 





. With a computer and modem 


Rating current support in 
science/technology as 
excellent or adequate 


State-certified library media 
specialist 


School characteristic 


Percent 


CV 

(percent) 


Percent 


CV 

(percent) 


Number 


CV 

(percent) 


School level 


Elementary 


28.3 


4.2 


54.2 


2.6 


41,911 


2.2 


Secondary 


48.2 


3.1 


49.3 


2.6 


23,287 


2.3 


Combined 


37.9 


7.9 


45.0 


6.7 


1,484 


6.0 


School size 


0-149 


29.5 


10.8 


50.2 


6.0 


4,411 


8.6 


150-299 


25.8 


8.5 


47.4 


5.9 


9,564 


5.7 


300-599 


31.4 


4.5 


50.8 


3.8 


25,137 


3.2 


600 or more 


44.6 


3.6 


59.7 


2.8 


23,279 


3.0 


Free lunch eligibility 


Less than 20% 


43.4 


4.8 


55.1 


4.2 


20,698 


4.0 


20-49% 


33.0 


4.5 


54.0 


3.5 


20,612 


3.8 


50% and more 


27.3 


6.6 


50.5 


4.0 


18,406 


4.0 



SOURCE: CVs are computed using estimates and standard errors from Chaney and Williams (1998). 



Table 7-8.- — Coefficients of variation (CVs) for selected items from the private school Library 
Survey, by selected school characteristics: Round 3 





With a computer and modem 


Rating current support in 
science/technology as 
excellent or adequate 


State-certified library media 
specialist 


School characteristic 


Percent 


CV 

(percent) 


Percent 


CV 

(percent) 


Number 


CV 

(percent) 


School level 


Elementary 


12.2 


12.3 


48.9 


3.5 


2,158 


10.5 


Secondary 


30.2 


6.3 


41.3 


7.5 


1,947 


8.5 


Combined 


27.9 


9.7 


31.2 


8.7 


1,266 


6.5 


School size 


0-149 


14.3 


15.4 


34.8 


8.0 


956 


15.2 


150-299 


17.3 


9.8 


46.5 


5.2 


1,363 


9.0 


300-599 


23.4 


10.3 


55.2 


4.5 


1,574 


11.6 


600 or more 


47.4 


6.5 


60.9 


5.6 


834 


7.1 


Catholic 


16.0 


8.1 


50.6 


3.4 


2,662 


6.0 



SOURCE: CVs are computed using estimates and standard errors from Chaney and Williams (1998). 
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technology as excellent or adequate, and the total number of state-certified library media 
specialists for public and private schools, respectively, classified by school characteristics. The 
tables also include the coefficients of variation (CVs) of these estimates computed using the 
bootstrap method of variance estimation. All but one of the estimates for the various categories 
of public school have CVs of less than 10 percent, and many of the CVs are much less than this. 
The CVs for the private school estimates are larger, with several exceeding 10 percent. 

Tables 7-9 and 7-10 present estimates and CVs from the Librarian Survey for public and private 
schools, respectively, classified by the same school characteristics. The three estimates in these 
tables are the percent of the librarians who are regular full-time librarians, the percent who are 
state -certified as library media specialists, and their mean number of years since they first 
worked as librarians. The CVs for the majority of the estimates for public school librarians are 
less than 5 percent and only one exceeds 10 percent. The CVs for the estimates for private 
school librarians are larger, but only 3 exceed 10 percent. 

7.10 EVALUATION OF ESTIMATES 

Dickson and Kindel (1998) compared estimates from the Library and Librarian Surveys with 
each other and with counts and estimates from the SASS School Survey, the TDSS, the CCD 
State Nonfiscal Survey, and the National Public Education Financial Survey (NPEFS). For each 
comparison, the authors developed a hypothesis as to whether the counts or estimates from the 
Library or Librarian Surveys should be higher, lower, or about the same as the counts or 
estimates from these other surveys. One of two outcomes resulted for each comparison — 
expectations were met or they were not. Comparisons were made by school type (public, 
private, or BIA) at the school, state, or national levels as appropriate. State-level comparisons 
were not available for private schools since the survey was not designed to produce such 
estimates. Items for comparison included the number of libraries, the role of librarians in school 
decisionmaking, and numbers of various types of library staff. 

Comparisons were made between the Library and Librarian Surveys at the school level for 
counts of state-certified librarians and for counts of professional staff members. Data were found 
to be consistent between these two surveys. Other results are presented first for libraries and 
then for librarians. 

Libraries. Most comparisons involving data from the Library Survey produced results consistent 
with the hypothesis, such as counts at the school level and estimates at the state and national 
levels of the number of libraries between the Library and School Surveys; counts at the school 
level and estimates at the state and national levels of total staff between the Library and School 
Surveys; estimates at the state and national levels of the number of Library Survey librarians and 
other staff working in the libraries of public schools compared to the number of librarians from 
the TDSS; estimates at the state and national levels of the number of total library staff between 
the Library Survey and the CCD State Nonfiscal Survey, and estimates at the state and national 
levels of the number of other professional staff in libraries from the Library Survey compared to 
the estimated number of library and media support staff from the CCD State Nonfiscal Survey. 
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Table 7-9.— Coefficients of variation (CVs) for selected items from the survey of public school 
librarians, by selected school characteristics; Round 3 .. 



School characteristic 


Regular full-time librarian at 
sampled school 


Certified by their state as a 
library media specialist 


Years since first worked as a 
librarian 


Percent 


CV 

(percent) 


Percent 


CV 

(percent) 


Mean 


CV 

(percent) 


School level 














Elementary 


62.4 


2.2 


79.4 


1.4 


11.7 


2.6 


Secondary 


76.2 


1.8 


90.8 


1.1 


14.8 


1.4 


Combined 


61.0 


.6.4 


87.5 


1.7 


13.6 


3.7 


School size 














0-149 


21.9 


14.2 


85.4 


3.3 


12.3 


5.7 


150-299 


36.6 


6.8 


81.9 


2.9 


11.7 


6.0 


300-599 


67.2 


2.1 


81.2 


1.7 


12.1 


3.3 


600 or more 


89.8 


1.5 


,85.0 


2.0 


13.8 


2.2 


Free lunch eligibility 














Less than 20% 


67.9 


2.9 


87.0 


1.6 


13.4 


2.2 


20-49% 


66.7 


3.4 


82.6 


1.8 


12.8 


3.1 


50% and more 


65.3 


3.7 


78.4 


2.6 


11.7 


2.6 


SOURCE: CVs are computed using estimates and standard errors from Chaney and Williams (1998). 






Table 7-10. — Coefficients of variation (CVs) for selected items from the survey of private school 


librarians, by selected school characteristics: Round 3 








Regular full-time librarian at 


Certified by their state as a 


Years since first worked as a 




sampled school 


library media specialist 


librarian 






CV 




CV 




CV 


School characteristic 


Percent 


(percent) 


Percent 


(percent) 


Mean 


(percent) 


School level 














Elementary 


40.8 


5.6 


22.2 


8.1 


9.2 


5.4 


Secondary 


69.8 


3.3 


47.8 


4.6 


11.7 


4.3 


Combined 


57.8 


6.9 


36.3 


9.1 


9.6 


6.3 


School size 














0-149 


. 75.2 


5.6 


33.2 


16.6 


10.8 


11.1 


150-299 


41.4 


5.6 


20.9 


8.6 


9.0 


5.6 


300-599 


67.1 


4.6 


36.1 


7.8 


9.9 


5.1 


600 or more 


86.4 


2.9 


58.9 


20.0 


12.8 


5.5 


Catholic 


54.5 


4.2 


30.0 


6.3 


10.0 


4.0 



SOURCE: CVs are computed using estimates and standard errors from Chaney and Williams (1998). 



A few comparisons did not produce the expected results. While the counts and estimates of the 
number of libraries were consistent between the Library and School Surveys, tables 7-11 and 
7-12 show that the reporting between surveys for schools with no libraries was less consistent, 
with the estimate from the Library Survey being lower than the School Survey estimate at the 
national level. 
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Table 7-11. — Estimates of public schools with no library media center from the Library Survey 
and the School Survey, by state: Round 3 



State 


Library 

Survey 


School 

Survey 


Library/ 

School 

(percent) 


State 


Library 

Survey 


School 

Survey 


Library/ 

School 

(percent) 


Total 


2,969 


3,833 


77 










Alabama 


0 


33 


0 


Montana 


0 


16 


0 


Alaska 


27 


48 


55 


Nebraska 


22 


19 


115 


Arizona 


43 


26 


166 


Nevada 


19 


10 


185 


Arkansas 


24 


0 


0 


New Hampshire 


17 


38 


44 


California 


365 


448 


81 


New Jersey 


169 


128 


131 


Colorado 


32 


25 


125 


New Mexico 


15 


34 


44 


Connecticut 


29 


51 


57 


New York 


142 


187 


76 


Delaware 


13 


11 


125 


North Carolina 


35 


42 


83 


District of Columbia 8 


7 


117 


North Dakota 


54 


43 


124 


Rorida 


28 


70 


40 


Ohio 


97 


87 


111 


Hawaii 


6 


6 


99 


Oklahoma 


88 


69 


128 


Idaho 


12 


30 


41 


Oregon 


8 


3 


233 


Illinois 


183 


219 


84 


Pennsylvania 


152 


181 


84 


Indiana 


43 


41 


104 


Rhode Island 


15 


17 


89 


Iowa 


27 


27 


100 


South Carolina 


16 


29 


55 


Kansas 


0 


46 


0 


South Dakota 


10 


15 


63 


Kentucky 


0 


26 


0 


Tennessee 


0 


47 


0 


Louisiana 


84 


76 


111 


Texas 


450 


311 


145 


Maine 


74 


55 


134 


Utah 


22 


23 


95 


Massachusetts 


169 


212 


80 


Virginia 


51 


64 


80 


Michigan 


33 


278 


12 


Washington 


60 


101 


59 


Minnesota 


44 


49 


89 


West Virginia 


176 


128 


137 


Mississippi 


68 


61 


111 


Wisconsin 


5 


20 


27 


Missouri 


18 


50 


36 


Wyoming 


21 


15 


142 


SOURCE: Dickson and Kindel (1998). 














Table 7-12.— 


Estimates of Bureau of Indian Affairs schools with no library media center from 




the Library Survey and the School Survey, by state: Round 3 


















Library/School 


State 




Library Survey 


School Survey 




(percent) 


Total 






10 




11 




90 


Idaho 






1 




1 




98 


Minnesota 






1 




1 




no 


New Mexico 






3 




3 




90 


South Dakota 




3 




4 




80 


Washington 






1 




1 




98 


Wisconsin 






1 




1 




93 



SOURCE: Dickson and Kindel (1998). 
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Although it was expected that counts and estimates of total staff would differ between the 
Library and School Surveys, with the counts from the Library Survey being higher because of 
the lack of instruction for reporting counts of staff assigned to more than one function or school 
for the Library Survey (instructions were present on the School Survey), there were some results 
contrary to this expectation when estimates for full-time and part-time staff were examined. It 
was found that the estimates of full-time library aides were notably lower on the Library Survey 
than those for the School Survey (see tables 7-13 and 7-14). A possible explanation was that in 
the School Survey imputation process, the TDSS information had precedence over information 
on the Librarian and Library Surveys during the first stage of imputation. In almost 500 
imputation cases, it was found that the LEA on the TDSS had reported no librarian, but the 
Library Survey reported one or more librarians. 

The comparison between the Library Survey estimate of state-certified librarians at the state and 
national levels and the CCD State Nonfiscal Survey counts of librarians and media specialists 
also contradicted the hypotheses. Table 7-15 shows that the Library Survey state estimates were 
generally higher than the CCD State Nonfiscal Survey counts. Possible reasons suggested were 
the different units of collection, different levels of reporting, and differences in terminology and 
definitions. 

The study recommended the following: review of the practice of classifying libraries as out of 
scope when their schools are out of scope; increasing response rates; and reviewing imputation 
and estimation procedures, definitions and forms design. 

Librarians. Most comparisons involving the Librarian Survey were consistent with the 
established hypotheses. These included comparisons at the state and national levels between the 
Librarian and School Surveys for full-time and part-time librarians; item comparisons at the 
school level between the Librarian and School Surveys about the role of librarians in schools 
regarding decisionmaking; national- and state-level estimates of the number of librarians 
between the public school Librarian Survey and the CCD State Nonfiscal Survey; and state- and 
national-level estimates of public school librarian salaries between the Librarian Survey and the 
NPEFS. 

The only comparison that did not conform to the hypothesis was the comparison of the Librarian 
Survey respondent’s main assignment at the school level to the number of full-time and part-time 
librarians from the School Survey. While there was a high level of agreement in reporting for 
full-time librarians for public and BIA schools, there were a notable number of cases where a 
full-time respondent on the Librarian Survey was reported as part time on the School Survey for 
private schools. For all types of schools, there were also a notable number of part-time librarians 
on the Library Survey reported as full time on the School Survey. Incomplete or confusing 
instructions for reporting part-time staff serving more than one function or school instructions for 
volunteers were believed to be important factors in these discrepancies. 

Dickson and Kindel proposed that response rates be improved; that data be collected and 
tabulated for volunteers; and that the imputation and estimation procedures, definitions, and 
forms design be reviewed. It was also suggested that data be collected in full-time equivalents. 
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Table 7-13. — Estimates of full-time library aides for public schools from the Library Survey and 
the School Survey, by state: Round 3 



State 


Library 

Survey 


School 

Survey 


Library/ 

School 

(percent) 


State 


Library 

Survey 


School 

Survey 


Library/ 

School 

(percent) 


Total 


26,555 


31,998 


83 










Alabama 


331 


532 


62 


Montana 


92 


168 


55 


Alaska 


79 


77 


103 


Nebraska 


305 


415 


73 


Arizona 


320 


540 


59 


Nevada 


70 


156 


45 


Arkansas 


122 


217 


56 


New Hampshire 


137 


181 


76 


California 


1,744 


1,944 


90 


New Jersey 


905 


722 


125 


Colorado 


568 


633 


90 


New Mexico 


217 


326 


67 


Connecticut 


476 


455 


105 


New York 


1,496 


1,675 


89 


Delaware 


17 


24 


73 


North Carolina 


538 


839 


64 


District of Columbia 


28 


40 


70 


North Dakota 


99 


116 


85 


Florida 


1,260 


1,412 


89 


Ohio 


1,187 


1,354 


88 


Georgia 


886 


1,284 


69 


Oklahoma 


605 


800 


76 


Hawaii 


56 


52 


108 


Oregon 


431 


588 


73 


Idaho 


100 


274 


37 


Pennsylvania 


845 


1,021 


83 


Illinois 


1,637 


1,471 


111 


Rhode Island 


59 


25 


233 


Indiana 


888 


1,101 


81 


South Carolina 


506 


817 


62 


Iowa 


464 


647 


72 


South Dakota 


101 


162 


62 


Kansas 


420 


545 


77 


Tennessee 


268 


312 


86 


Kentucky 


290 


532 


55 


Texas 


2,636 


3,319 


79 


Louisiana 


256 


219 


117 


Utah 


129 


166 


78 


Maine 


231 


403 


57 


Vermont 


91 


122 


75 


Maryland 


342 


467 


. 73 


Virginia 


793 


710 


112 


Massachusetts 


576 


507 


114 


Washington 


556 


630 


88 


Michigan 


1,166 


1,100 


106 


West Virginia 


19 


41 


47 


Minnesota 


726 


705 


103 


Wisconsin 


652 


971 


67 


Mississippi 


404 


463 


87 


Wyoming 


115 


165 


69 


Missouri 


320 


555 


58 











NOTES: For this table, library media center aides are defined as: For the Library Survey = library aides and clerks working in the library. For 
the School Survey = library aides. 

SOURCE: Dickson and Kindel (1998). 



Another study by Cole (1997) estimated the possible bias in average librarian salary as collected 
in the Librarian Survey, for both public and private schools. Only one library employee per 
school (presumably the head librarian) completed the Librarian Survey questionnaire, with only 
this respondent’s income reported. Because some schools have more than one librarian and the 
respondent head librarian’s salary would generally be higher than those of the library’s other 
employees, it was felt that the average librarian salary estimate from the Librarian Survey was an 
overestimate of the average salary of all librarians in the country. The study concluded that the 



Table 7-14. — ^Estimates of full-time library aides in Bureau of Indian Affair schools from the 
Library Survey and the School Survey, by state: Round 3 



State 


Library Survey 


School Survey 


Library/School 

(percent) 


Total 


44 


53 


83 


Arizona 


12 


18 


67 


California 


1 


1 


136 


Connecticut 


0 


0 


0 


Florida 


0 


0 


0 


Idaho 


0 


0 


0 


Iowa 


0 


0 


0 


Kansas 


0 


0 


0 


Louisiana 


0 


0 


0 


Michigan 


1 


0 


0 


Minnesota 


1 


1 


115 


Mississippi 


0 


0 


0 


Montana 


0 


0 


0 


Nevada 


0 


0 


0 


New Mexico 


15 


14 


104 


North Carolina 


0 


. 2 


0 


North Dakota 


3 


4 


83 


Oklahoma 


1 


0 


0 


Oregon 


0 


0 


0 


South Dakota 


7 


10 


64 


Utah 


0 


0 


0 


Washington 


3 


2 


134 


Wisconsin 


0 


1 


0 


Wyoming 


' 0 


0 


0 



NOTES: For this table, library media center aides are deEned as: F6r the library Survey = library aides and clerks working in the library. For 
the School Survey = library aides. 

SOURCE: Dickson and Kindel (1998). 

original salary estimates were close to those produced in this study. The average bias was 
estimated at 1 percent for public school librarians and 4 percent for private school librarians. 
The positive average bias results mean that the original estimates were overestimates as 
expected. For about 80 percent of public school libraries and 70 percent of private school 
libraries, the bias in the original estimate of average salary was zero (includes libraries with only 
one employee). Of the 20 percent of public school and 30 percent of private school libraries with 
nonzero bias values, about 5 percent of public and 6 percent of private school libraries had bias 
values that indicated that the original average salary estimates overestimated the “true” average 
salaries by an average of 50 percent or more. 
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Table 7-15. — Estimates of public school librarians from the Library Survey with CCD State 
Nonfiscal Survey, by state: Round 3 



State 


Library 

Survey 


CCD 

Survey 


Library/ 

CCD 

(percent) 


State 


Library 

Survey 


CCD 

Survey 


Library/ 

CCD 

(percent) 


Total 


66,682 


50,501 


132 










Alabama 


1,247 


1,194 


104 


Montana 


896 


339 


264 


Alaska 


253 


156 


162 


Nebraska 


973 


558 


174 


Arizona 


825 


773 


107 


Nevada 


281 


239 


118 


Arkansas 


1,101 


948 


116 


New Hampshire 


301 


254 


119 


California 


2,609 


928 


281 


New Jersey 


2,252 


1,800 


125 


Colorado 


■ 754 


704 


107 


New Mexico 


363 


243 


149 


Connecticut 


707 


654 


108 


New York 


3,987 


2,983 


134 


Delaware 


143 


116 


124 


North Carolina 


2,203 


2,151 


102 


District of Columbia 


157 


173 


91 


North Dakota 


459 


179 


256 


Florida 


2,370 


2,513 


94 


Ohio 


2,650 


1,776 


149 


Georgia 


1,969 


2,053 


96 


Oklahoma 


1,430 


860 


166 


Hawaii 


230 


286 


80 


Oregon 


803 


665 


121 


Idaho 


321 


168 


191 


Pennsylvania 


3,121 


2,170 


144 


Illinois 


2,936 


1,973 


149 


Rhode Island 


292 


82 


356 


Indiana 


1,458 


1,047 


139 


South Carolina 


1,168 


1,085 


108 


Iowa 


1,354 


652 


208 


South Dakota 


579 


208 


278 


Kansas 


1,504 


973 


155 


Tennessee 


1,514 


1,280 


118 


Kentucky 


1,420 


1,187 


120 


Texas 


5,101 


4,143 


123 


Louisiana 


1,240 


1,217 


102 


Utah 


404 


267 


151 


Maine 


385 


241 


160 


Vermont 


284 


199 


143 


Maryland 


1,239 


1,078 


115 


Virginia 


1,783 


1,945 


92 


Massachusetts 


1,161 


567 


205 


Washington 


1,339 


1,255 


107 


Michigan 


2,022 


1,500 


135 


West Virginia 


550 


360 


153 


Minnesota 


1,430 


984 


.145 


. Wisconsin 


1,911 


1,339 


143 


Mississippi 


781 


640 


122 


Wyoming 


292 


135 


217 


Missouri 


2,130 


1,261 


169 











NOTES: The CCD state nonfiscal survey collects data in full-time equivalents, while the Library Sui^ey collects counts. For the Library Survey, 
the data include those working vyith computer resources. For the CCD Survey, the data may include library media specialists/ librarians providing 
district and regional level media services. 

SOURCE: Dickson and Kindel (1998). 

It was recommended that in future data collections, either the salaries of all library employees be 
obtained, or a more accurate way be developed to account for the salaries of nonhead librarians 
at libraries with more than one employee. 
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8. THE STUDENT RECORDS SURVEY 



8.1 INTRODUCTION 

A Student Records Survey was added to the Schools and Staffing Surveys (SASS) for the 1993- 
94 Round (Round 3). The motivation for this survey was that student information, combined 
with information from the teacher and school samples, would provide the opportunity to explore 
student assignment and behavior patterns over a wide range of teachers and schools (Ingersoll, 
1995). In particular, the purpose of the Student Records Survey was to collect data that could .be 
used to examine the distribution of school programs and the quality of teachers among students 
of differing demographic and academic characteristics and to describe students' participation in 
school programs and services. A topic of particular interest was the participation of American 
Indian students in special programs. Data for this survey were collected from administrative 
records, not directly from students; such data items as sex, date of birth, race/ethnicity, country 
of birth, language spoken at home, grade level, participation in programs and services, grade 
point average (GPA), and attendance record are readily available from such records. The survey 
included students in public schools, private schools, and Bureau of Indian Affairs (BIA) schools, 
and was designed to produce estimates at the national level for students from each of these 
school types. 

This chapter is organized in the same manner as the previous chapters — that is, with a section for 
each major survey activity: sampling frames (8.2), sample design (8.3), content (8.4), data 
collection procedures (8.5), data processing (8.6), measurement error (8.7), nonresponse (8.8), 
and estimation (8.9). Since no evaluation of estimates was available, the chapter contains no 
section on this topic. 

8.2 SAMPLING FRAMES 

The target population for the 1993-94 Student Records Survey consisted of all students included 
in grades K-12 in the fall of 1993 who were in the target population of schools for the School 
Survey. Furthermore, since the survey was to be linked to the Teacher Survey, the students in 
the target population also needed to be taught by one or more teachers in the target population 
for the Teacher Survey. 

The sampling frame for the Student Records Survey was created in four stages corresponding to 
the four stages of sampling involved. At the first stage, a sample of schools was selected for the 
School Survey from the school sampling frames. The properties of those frames are described in 
sections 2.4.1 and 2.4.2. At the second stage, a subset of the schools sampled for the School 
Survey was selected for the Student Records Survey. At the third stage, a sample of teachers 
was selected in this subset of schools from teachers selected for the Teacher Survey. At the 
fourth stage, a sample of students of selected teachers was selected from class rosters. Some 
problems that were encountered with the class rosters are described below. 
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8.3 SAMPLE DESIGN 



The first stage of sampling for the Student Records Survey was the selection of the schools for 
the School Survey, as described in section 2.4. The second stage involved the selection of a 
subsample of the schools in the School Survey. This subsample comprised all BIA, Native 
American, and Alaskan schools, together with subsamples of the remaining public and private 
schools. The subsample of schools was designed to minimize the amount of overlap with the 
sample of schools selected for the Library and Librarian Surveys (Abramson et al., 1996). The 
rest of the public schools were stratified by grade level and local education agency (LEA) 
urbanicity and then sorted within each explicit stratum by 1993-94 SASS school stratum. Census 
Region, SASS order of selection code, and SASS School Common Core of Data Identification 
(CCD ID). Private schools were stratified by recoded affiliation (Catholic, other religious, 
nonsectarian) and grade level, and then sorted by frame (list frame, area frame) and school 
enrollment within each explicit stratum. Schools were subsampled for the Student Records 
Survey using probability proportional to size (PPS) sampling. The measure of size used in the 
subsampling was the school's measure of size for the School Survey multiplied by the school's 
inverse probability of selection for that survey. The use of this measure of size in effect cancels 
out the differential sampling rates used for sampling schools in different strata for the School 
Survey. Any school with a measure of size larger than the sampling interval was selected with 
certainty for the student sample. 

The next stage of sampling selected approximately three teachers from each subsampled school. 
Within the selected schools, all teachers sampled for the Teacher Survey were eligible for the 
Student Records Survey, and three of them were randomly selected per school. If three or fewer 
teachers in a selected school were sampled for the Teacher Survey, then all sampled teachers 
within the school were selected. 

The final stage of sampling was to select two students per selected teacher, for a total of six 
students per selected school. This selection was done by U.S. Census Bureau staff in 
Jeffersonville through telephone contact with a representative of the sampled school, when 
possible, and through a personal visit of a field representative when the selection could not be 
done by telephone. The first step was to determine teacher eligibility. Sampled teachers who did 
not teach regularly scheduled classes were considered ineligible and were excluded. Eligible 
teachers were then classified as either self-contained (teaching the same group of students for 
most of the day) or departmental. For self-contained teachers, a copy of the class roster was 
requested. Two students were selected randomly from the roster by U.S. Census Bureau staff. 
The procedure was more complicated for departmental teachers. For such teachers, a sample 
class period was selected first for each teacher. To accomplish this, a set of five possible class 
periods was selected for each school (one from each day of the week). Next, it was determined 
during which of the selected five class periods the departmental teacher taught a class. From 
these, one class period was selected randomly. If no class period was found for the departmental 
teacher from the first five selected, then five more class periods were selected, and the process 
was repeated. If this still did not prove successful in selecting a class, the school was asked for 
all class periods that the teacher taught and one was randomly selected. A copy of the class 
roster for the sample period and day was requested and two students were selected by U.S. 
Census Bureau staff. 
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The Student Records Survey selected a total sample of 1,751 schools, allocated across types of 
schools as shown in table 8-1. The table also shows the resulting numbers of teachers and 
students in these schools sampled according to the procedures outlined above. The total sample 
size of 6,933 students consisted of 5,697 public school students and 1,236 private school 
students. To meet the survey's analytic objectives, all BIA, Native American, and Alaskan 
schools were sampled, and thus the student sample contained a large sample of students in 
Native American schools, and oversamples of Alaskan students and students in BIA schools. 

Table 8-1. — Numbers of schools, teachers, and students in the Student Records Survey, by 
sector: Round 3 



Type of school 


Number of schools 


Number of teachers 


Number of students 


Total 


1,751 


4,651 


6,933 


Private 


381 


903 


1,236 


Total public 


. 1,370 


3,748 


5,697 


BIA 


176 


430 


602 


Native American 


444 


1,262 


2,024 


Alaskan 


199 


549 


759 


Other public 


551 


1,507 


2,312 



SOURCE: Gruber, Rohr, and Fondelier (1996). 



Problems encountered during student sampling. In addition to problems of nonresponse from 
schools refusing to participate in sampling and failing to complete the student records 
questionnaires, Parmer, Abramson, Cole, Colaciello, and Garrett (1997) outline a number of 
problems that occurred as a result of the fact that three of the four stages of selection for students 
relied on information provided by the respondent (school principal or other school staff member 
who is familiar with the student's school records). Problems included schools having difficulty 
in following the sampling instructions over the telephone when having to go through a maximum 
of three different sets of classes for up to three teachers to identify one eligible class per teacher; 
finding some unrealistic values for student sampling information on some student records 
questionnaires, suggesting the possible misreporting of information; the erroneous classification 
of some sampled teachers as out of scope; and the fact that the number of classes that students 
take was not asked. As a result of these problems, information needed to select the student 
sample was sometimes missing or inaccurate, and difficulties arose in computing the students' 
probabilities of selection (King and Kaufman, 1994). For example, the lack of information on 
the numbers of classes taken by sampled students had to be handled in developing the weights by 
assuming that all students in a school took the same number of classes. 

Parmer et al. (1997) suggest that an automated sampling worksheet with internal edits would 
eliminate many of these sampling problems.' They also recommend that the selection of classes 
should be made easier in the future and that information should be collected on the number of 
classes taken by each sampled student. Cole and Schwanz (1998) also recommend for the future 
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the development of clear guidelines for determining teacher eligibility in order to reduce 
misclassification errors. 

8.4 CONTENT 

The Student Records Survey questionnaire covered the following topics: 

0 The student's demographic background; 

a Whether the student had ever dropped out of school or been suspended from school; 
s School programs/services that the student currently receives; 
o Current disabilities of the student; 

□ Counseling services that the student is currently receiving; 

o Number of days the student was absent during a given time period; 

□ Whether the student had ever been retained in a grade; 

□ The student's current grade level; 

□ Information on the type of science and math courses enrolled in (if in grade 9 or above); 

0 The number of advanced placement courses the student was enrolled in (if in grade 9 or 
above); and 

0 The student's GPA (if in grade 9 or above). 

There was only one version of the Student Records Survey questionnaire, which was sent to all 
sampled schools to complete for their sampled students. 

8.5 DATA COLLECTION PROCEDURES 

Field test for 1993. In spring 1993, a field test of the Student Records Survey was conducted to 
determine whether collecting student data from school records was feasible. The field test tested 
the proposed student records questionnaire, procedures for selecting the student sample, and the 
method of data collection. 

Since it was planned that students would be sampled for the Student Records Survey from the 
class rosters of sampled teachers and since the students could each be listed on several rosters, 
they could have multiple routes of selection for the sample. Information on this multiplicity is 
needed since it affects the students' selection probabilities and hence their basic weights. To 
obtain some information on this multiplicity, questions were developed to determine the numbers 
of the class rosters of the sampled teachers on which each sampled student was listed. Prior to 
the field test, cognitive research was conducted in four schools to test' one method of obtaining 
this information. Based on the results of that research, the questions were revised and a new set 
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of questions was tested in two more schools. The set of questions was revised again to produce 
the set used in the field test. 

The Student Records Survey field test tested two alternative methods of data collection — a 
mailout of the student records questionnaire with telephone followup and a personal visit to 
select the student sample and collect the student records data. In the first method, schools were 
mailed a Teacher Listing Record on which they were asked to list their teachers. Three teachers 
were selected from each school's list. Schools were recontacted to ask for a class roster for a 
specified class period for each selected teacher, with a request that the rosters be sent to the U.S. 
Census Bureau by mail or fax. Three students were then selected from each roster, for a total of 
nine students from each school. Although a few schools did provide rosters by fax, the majority 
had not returned the rosters by the time the student sample had to be selected. Most of the 
student sampling was therefore done over the telephone by clerks at a U.S. Census Bureau 
processing center. Some schools (34 public and 44 private) refused to provide students' names 
over the telephone and used initials or code numbers instead. Mailout of student records 
questionnaires, with the school's name and nine students' names on the front of the student 
records questionnaire, occurred in April 1993. If a completed student records questionnaire was 
not received from a school, telephone followup was conducted. Table 8-2 shows the numbers of 
public and private schools selected, the numbers of schools that provided a Teacher Listing 
Record, the numbers of schools that provided a class roster, and the numbers of completed 
student records questionnaires for the mail component (i.e., mailout of the student records 
questionnaires) of the field test. Only 76 percent of the public schools and 69 percent of the 
private schools in the mail component of the field test returned the Teacher Listing Records. 
Further losses occurred at the stages of providing class rosters and student records 
questionnaires, so that overall only 65 percent of public schools and 53 percent of private schools 
provided the survey data. 



Table 8-2. — Numbers of schools participating in the mail component of the Student Records 
Survey field test, by sector and survey stage; Round 3 



Category 


Public schools 


Private schools 


Selected 


282 


194 


Returned Teacher Listing Record 


213 (76%) 


133 (69%) 


Provided rosters 


204 (72%) 


131 (68%) 


Completed student records 
questionnaires 


184 (65%) 


103 (53%) 



SOURCE: Gruber et al. (1996). 



Problems occurred when schools would provide only student initials or code numbers. When the 
student records questionnaires arrived at the schools, there were only nine sets of numbers or 
initials on the cover with no indication of the teacher or class involved. In some cases, schools 
were able to resolve the problem by obtaining information on the teachers and class periods for 
the sampled students from the U.S. Census Bureau. However, it did not prove possible to handle 



these cases when the school did not return the mail student records questionnaire and followup 
by field representatives was required. 

The personal visit method of data collection in the field test was conducted in April 1993 and 
included 23 public schools and 5 private schools, most of which were BIA or Indian schools. All 
5 private schools and 18 of the public schools agreed to participate. The student sample was 
selected and the student records questionnaires were completed during a personal visit by a U.S. 
Census Bureau field representative. Since the Department of Education and BIA had requested 
the inclusion of some supplemental questions in the Student Records Survey for American 
Indian and Alaska Native students, the student records questionneiire used in this part of the field 
test included a one-page Indian supplement to test the feasibility of collecting these data. 

The field test results led to the Student Records Survey questionneiire being totally redesigned for 
the 1993-94 survey. The 32-page student records questionneiire with the nine students’ names on 
the cover used in the field test was replaced by a separate four-page student records 
questionnaire for each student. The student records questionnaires were stapled together inside a 
cover folder that was preprinted with the school's control number, name, address, and the number 
of students selected from the school. To mainteiin better control of the siample, each sample 
student was assigned his/her own control number whereas, in the field test, only the school had a 
control number; also included on the cover information for each four-page student records 
questionnaire was the student's name, teacher's name, selected class period and the names of the 
three teachers selected for the Student Records Survey. 

Procedures. In December 1993, an advance letter was mailed to each of the 1,751 schools 
selected for the Student Records Survey to describe the purpose of the survey and to expleiin that 
the student information was to be collected from administrative records — not from the students 
themselves. The letter also notified the school that a U.S. Census Bureau representative would 
contact the school by telephone in January or early February to select a student sample. 

Teacher lists had previously (in September 1993) been requested from all schools sampled for 
Round 3 of the School Survey. Teachers were sampled for the Teacher Survey from these lists 
as described in section 5.4.2. Within each school sampled for the Student Records Survey, a 
subsample of three of the teachers sampled for the Teacher Survey was selected for the Student 
Records Survey. If a school had three or less sample teachers, all sample teachers from the 
school were selected. 

After the sample of three teachers had been selected, the next stage in the operation was for U.S. 
Census Bureau staff to call the schools to select a class period for each teacher, to ask the school 
to obtain student rosters for those classes, and to select two students from each class. When 
schools would not allow sample selection by telephone (288 schools), a personal visit to the 
school was arranged to select the student sample and to complete the student records 
questionnaires. These personal interviews occurred between March 21 and June 3, 1994, and 
data from student records were collected for 811 of the 872 students selected in this way. 

Packages of student records questionnaires were mailed in March 1994 to the sampled schools 
for which student selections had been made. A second mailing of 68 percent of the student 
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records questionnaires took place in April to schools that had not responded by that time. After 
that, any school not returning its student records questionnaires was called by a field 
representative from one of the 12 U.S. Census Bureau regional offices, who completed paper 
copies of the student records questionnaires by telephone. Telephone followup occurred for 
2,704 student records questionnaires (32 percent), of which 1,650 (61 percent) were completed. 

8.6 DATA PROCESSING 

In data processing, separate records were created for each sampled student. Of the 121 variables 
(items that required a response) that were computer edited, 20 had change rates greater than 10 
percent (where the change rate is the number of item changes divided by the number of 
completed student records questionnaires). These 20 items included 14 of a set of 15 items that 
were included to check that the selected student was taught by the selected teacher and during the 
class period that had been selected. Five of the other six problematic questions related to 
students in grades 9 or above. They were about the number of advanced placement courses 
completed or currently enrolled in, the student's current GPA and the grade scale used to 
compute it, and the number of transcripts sent to colleges or universities for the student. 

8.7 MEASUREMENT ERROR 

A reinterview program was conducted to assess the reliability of response for a selection of items 
included in the Student Records Survey. The student records reinterview questionnaire asked 
about whether the student was taught by the selected teacher, the student's race/ethnicity, grade 
level, services the student receives, and the student's enrollment in a mathematics class. All the 
questionnaire data were collected by mail. A sample of two students was selected from each 
school that responded to the Student Records Survey from amongst the students selected for that 
survey. This sample was selected on a flow basis as the original questionnaires were returned. 
Of the students selected in this manner, data were obtained for 1,243. Results of the reinterview 
were used to validate the information required for deriving basic student weights. 

8.8 NONRESPONSE 

Unit nonresponse. Unit nonresponse in the Student Records Survey could occur because the 
sampled school did not participate in the student sampling (by providing a teacher list and class 
rosters for sampled teachers) or because the school failed to return the completed student records 
questionnaires for sampled students. The response rates for participating in the student sampling 
were 88 percent for public schools, 79 percent for private schools, and 94 percent for Indian 
schools. In schools participating in the student sampling, the student records response rates were 
90 percent in public schools, 88 percent in private schools, and 94 percent in Indian schools. 
Thus the overall weighted unit response rates for the Student Records Survey were 80 percent in 
public schools, 70 percent in private schools, and 88 percent in Indian schools. 

As was done for other Round 3 SASS surveys, Monaco, Salvucci, Zhang, Hu, and Gruber (1998) 
conducted analyses to examine variation in response rates in the Student Records Survey by 
school characteristics. The analyses relate to the conditional student response rates in schools 
participating in the student sampling. Table 8-3 summarizes the results for both public and 
private school students. For students in public schools, the only characteristic for which the 
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Table 8-3. — Weighted response rates for the Student Record Survey, by selected school 
characteristics: Round 3 





Weighted response rate (percent)* 


Characteristics 


Public Private 



Region: 



West 


83 


93 


South 


93 


82 


Northeast 


91 


88 


Midwest 


97 


93 




P=0.003 


P=0.128 


Urbanicity: 


Central city 


95 


92 


Urban fringe/large town 


88 


83 


Rural 


92 


92 




P=0.077 


P=0.142 


School level: 


Combined 


95 


82 


Secondary 


90 


89 


Elementary 


92 


91 




P=0.581 


P=0.131 


School size: 


1-149 


98 


84 


150-499 


92 


91 


500-749 


94 


78 


750+ 


89 


95 




P=0.272 


P=0.137 


School type: 


Nonregular 


96 




Regular 


91 

P=0.402 




Minority enrollment: 


<5.5% 


93 




5.5-20.5%. 


89 




20.5-50.5% 


92 




>50.5% 


91 






P=0.612 




School sampled with certainty: 


Yes 


96 




. No ■ 


91 

P=0.192 




School sampled in 1990-91: 


Yes 


90 


96 


No 


92 


86 




P=0.650 


P=0.025 


Typology: 


Catholic 




93 


Other religious 




85 


Nonsectarian 




79 

P=0.041 





’The basic weight is used to reflect the probability of selection. 
^Not applicable. 

SOURCE: Monaco et al. (1998). 
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conditional response rates differed significantly by level (at the 5 percent significance level) was 
region; the response rate was highest in the Midwest and lowest in the West. For private school 
students, only two variables, whether or not the school was sampled in 1990-91 and typology, 
had significantly different conditional response rates by level. Students in private schools also 
sampled in Round 2 had much higher response rates than students in private schools not sampled 
in the previous round, but this finding may be explained by the overlap sampling procedures 
used, with overlap minimized for association groups with low response rates in the previous 
round. Students in Catholic schools had an appreciably higher response rate than students in 
other religious schools, who in turn had a higher response rate than students in nonsectarian 
schools. 

A second part of the Monaco et al. (1998) study examined the association of response status for 
the various SASS components. For public schools, response to the Student Records Survey was 
associated only with the LEA response to the Teacher Demand and Shortage Survey. For private 
schools, response to the Student Records Survey was associated only with response to the School 
Survey. 

Item nonresponse. Item response rates in the Student Records Survey were generally high, with 
only a few instances of rates less than 95 percent for any of the three sectors (public, private, and 
Indian schools). Only three items had a consistently sizable level of nonresponse: an item 
asking whether the student had been suspended or expelled from this school due to alcohol or 
drug abuse had the highest level of nonresponse (around 10 percent); a "mark all that apply" item 
asking about the grades in which the student had been retained, asked only for students who ever 
had been retained (around 8 percent for public and private school students, 20 percent for Indian 
school students); and an item asking about days absent during the last completed grading period 
(around 7 percent). 

8.9 ESTIMATION 

Imputation. Missing values were imputed by one of three methods for all but the first three 
student records questionnaire questions, which were asked to verify sampling information. The 
first method used other available information from the same record or by making assumptions 
about the intended answer. The second method used was the hot deck method. Finally, a 
selected group of questions — gender, grade in which student was retained, mathematics courses, 
and science courses — were always imputed clerically. In a very few cases, some other entries 
also were clerically imputed. Imputation flags were used to denote the stage of imputation as for 
all other SASS surveys. 

Imputations were also carried out as a last resort when student sampling information was missing 
from student data records (Parmer et al., 1997). The item requesting the number of times the 
student was taught by teacher each week was the least imputed, at a rate of about 2 percent. The 
highest overall imputation rate, at almost 6 percent, was recorded for the item requesting the 
class size for the selected class period. (Cole and Schwanz, 1998) 
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Weighting. The final weight for students from public and private schools consists of the product 
of the following factors: 

■ Basic weight: the inverse of student's probability of selection conditioned on the specific set 
of sample teachers selected for the student sample at the school; 

■ School nonresponse adjustment factor: an adjustment to account for schools that did not have 
students selected because the school did not respond to either the teacher or student sampling 
procedures; 

■ Misclassified teacher adjustment factor: an adjustment to account for sampled teachers 
reported to not be teaching regularly scheduled classes during student sampling, but later 
reported to be teaching in the Teacher Survey; 

■ First-stage ratio adjustment factor: an adjustment to make the sample estimates conform to 
known frame totals of the number of students; 

■ Student noninterview adjustment factor: an adjustment to account for sampled students 
whose schools did not return questionnaires or returned incomplete questionnaires, and 

■ Student adjustment factor: a factor to adjust for the inconsistency between the estimated 
number of students from the SASS school data files and the SASS student sample files. 

Different adjustment cells were used for public, private, and BIA school students. 

Student weighting problems. As described earlier, students were sampled within schools by first 
sampling teachers, then if necessary sampling a class period for a sampled teacher, and finally 
sampling two students from the roster for the sampled class. With this procedure, there are 
problems in determining a student's probability of selection and hence the basic weight. A 
student's selection probability depends on the number of different classes he or she took, on the 
sizes of those classes, and on the selection probabilities for the associated class periods. Lacking 
full information on these factors, approximations had to be used for the selection probabilities 
and for the basic weights. 

A problem was discovered for American Indian students in public schools, where the estimated 
coefficient of variation of the estimated total number of students was extremely high, above 50 
percent. This large coefficient of variation occurred despite the oversampling of these students. 
The oversampling was accomplished by selecting BIA schools and schools with large percents of 
American Indian students with certainty or with high probability. However, a sizable proportion 
of American Indian students attend other public schools and hence were not oversampled. 
American Indian students selected from these other schools therefore received very high weights 
relative to those sampled from the oversampled schools, and these high weights led to the low 
precision of the estimates for American Indian students. Parmer et al. (1997) investigated 10 
alternative weighting schemes for addressing this problem. The scheme finally chosen for the 
American Indian student sample involved truncating the weights, expanding the acceptable range 
for the student adjustment factor, and changing the order of collapsing adjustment cells for the 
student adjustment factor. 
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Variance estimation. In Round 3, a bootstrap method of variance estimation was employed for 
all SASS surveys except the Student Records Survey. For the Student Records Survey, the 
balanced half-sample replication procedure was used because of the small size of the student 
records sample. For sample students in schools not selected with certainty for this survey, 
variance strata were formed by pairing the schools. For sample students in schools selected with 
certainty, sample teachers were used to define the variance strata. A total of 48 replicates was 
used (Abramson et al., 1996). 
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9.1 INTRODUCTION 



The earlier chapters in this report have discussed the quality of the data produced by the various 
surveys that have constituted the Schools and Staffing Survey (SASS) in the first three rounds. 
The focus of these chapters has been on potential sources of error — in particular, noncoverage, 
unit and item nonresponse, measurement error, and sampling error — and their possible impact on 
the accuracy of the survey estimates. This chapter has two objectives. First, it broadens the 
concept of quality to include issues of relevance, accessibility, and timeliness (section 9.2). 
Second, it attempts to synthesize the findings from earlier chapters to identify key areas where 
efforts for methodological improvements might be most effectively directed and where further 
information is needed for the assessment of survey quality (section 9.3). The chapter ends with 
some concluding remarks in section 9.4. 

9.2 RELEVANCE, ACCESSIBILITY, AND TIMELINESS 

The ultimate goal of conducting SASS is to provide the data required by policymakers and 
researchers to understand the characteristics of the U.S. elementary and secondary education 
system. Thus, the goal of SASS is to collect the relevant data, to make the results and the data 
files readily accessible, and to provide data that are as up to date as possible. This section 
describes the procedures adopted in SASS on the quality dimensions of relevance, accessibility, 
and timeliness. 

Relevance. SASS attempts to ensure that it collects the data needed to inform policy decisions 
and stimulate research by maintaining close contacts with the broad user community. Before 
each administration of the survey, the National Center for Education Statistics (NCES) enlisted 
the help of many experts and specialists in the educational and policy communities to examine 
SASS and propose changes to the survey's content and methods. In addition, the Advisory 
Council on Education Statistics (ACES) — the advisory panel for NCES — reviewed the plans for 
SASS at each round, and the SASS Technical Review Panel met regularly to discuss the 
recommendations made by other groups and to provide a broad evaluation of the plans for the 
content, design, analysis, and reporting of the survey. 

As discussed in section 1.2, NCES introduced SASS in 1987 in the context of the national school 
reform movement of that period. There was an acute need for data on the teacher labor market, 
sources of new teachers, and the conditions under which teachers would choose to enter, remain, 
or leave teaching. Teacher quality and school characteristics were issues of concern. With the 
help of experts and specialists in the educational research and policy community, the content of 
the initial round of SASS, therefore, was developed to measure critical aspects of teacher supply 
and demand, to profile the qualifications and working conditions of teachers and administrators, 
and to describe basic conditions in schools as workplaces and learning environments (see 
Hudson and Darling-Hammond, 1987). 

In planning for Round 2, attention focused on the need for significant changes in a number of the 
operational and design elements, as well as on improving the wording and design of specific 
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questions, but the overall content of Round 1 was seen as continuing to meet the data needs. The 
wording of selected questions and the design of questionnaires underwent considerable 
modification between Rounds 1 and 2, but the basic subject content remained essentially 
unchanged. 

Again, the planning for Round 3 focused primarily on further changes in design and procedures, 
such as modifying questionnaires to simplify the work of respondents, deleting some questions 
that yielded little usable information, and reformatting others. However, some additions and 
modifications to content were included in Round 3. For example, a student record questionnaire, 
designed to obtain student data from school records, became a new component of SASS and, in 
Round 3, the data collection was expanded to include a component on libraries and librarians. 

The 6-year period between Rounds 3 and 4 provided the opportunity for a major review of the 
content and purposes of SASS, in light of the many changes in education policy and thinking 
since the inception of the survey in 1987. The redesign of SASS engaged many segments of the 
education research and policy community. In 1995, NCES commissioned experts from 
academia, the research community, and specialists in technology, teacher education, and state 
and local data collection to examine SASS and propose changes in content and methodology 
(Mullens and Kasprzyk, 1997). This was followed by expert panels composed of researchers, 
policy analysts, and others involved in school reform, who provided comments and 
recommendations for the new round of data collection. Separate panels explored how SASS 
could contribute to a better understanding of teacher supply and demand and school reform. 
They reviewed the previous SASS content on these topics and identified common themes or 
issues that suggested the need for revisions. The SASS Technical Review Panel met regularly to 
discuss the recommendations proposed by other groups, and it provided a broad evaluation of the 
plans for the context, design, analysis, and reporting of SASS data. NCES's advisory panel, 
ACES, also reviewed these and similar issues in preparation for a refocused SASS in 1999-00. 

Round 4 of SASS that emerged from these diverse efforts shifts emphasis from teacher supply 
and demand issues to the measurement of teacher and school capacity, both objectives of the 
recent school reform agenda. These changes reflect shifts in the policy debate from teacher 
quantity — the numbers of vacant teaching positions — to teacher quality — the qualifications of 
teachers who were hired and retained. To measure teacher capacity, the redesigned SASS 
examines teacher qualifications, teacher career paths (including induction experience), and 
professional development. To measure school capacity, SASS concentrates on school 
organization and decisionmaking, curriculum and instruction, parental involvement, school 
safety and discipline, and school resources. 

Accessibility. The value of a survey depends on the extent to which its data are used, which in 
turn depends on the accessibility of the survey results and the survey data files. Moreover, the 
proper use of the survey data requires the availability of good documentation. 

Results from SASS are published in descriptive reports, analytic reports, and issue briefs. The 
descriptive reports, consisting of statistical profiles, education tabulations (ED tabs), and survey 
reports, present basic information about schools, principals, and teachers; for example, the length 
of teaching experience of principals, student participation in public school programs, such as free 



or reduced price lunches, and the qualifications and years of experience of public and private 
school teachers (e.g., Fiore and Curtin, 1997; Bobbitt, Broughman, and Gruber, 1995; Henke, 
Choy, Geis, and Broughman, 1996). Analytic reports examine issues of particular interest in 
more detail, for instance, teacher attrition, teacher and principal qualifications, professional 
development, out of field teaching, and sources of supply for new teachers (e.g., Ingersoll, Han, 
and Bobbitt, 1995; Ingersoll and Gruber, 1996). The issue briefs provide short accounts of about 
two pages in length on topics of current concern, and present limited data. A complete list of 
SASS publications, with instructions for how to view, order, or download them, is available on 
the World Wide Web at http://nces.ed.gov/surveys/sass; 

NCES recently conducted a study to explore the satisfaction of key customers with SASS 
publications by means of individual interviews with 30 selected representatives from state 
education agencies and 19 other key customers (Rouk, Weiner, and Riley, 1999). In general, 
these customers considered the publications to be easily accessible, the content appropriate for 
their data needs, and the presentations quite clear. In addition, focus group discussions were held 
with individuals from Federal and state government, research, and data management 
organizations to obtain reactions concerning the appropriateness, usability, and accessibility of 
two key SASS publications: A Statistical Profile, 1993-94 (Henke et al, 1996) and SASS by State, 
1993-94 (De Mello and Broughman, 1996). In general, the comments of the participants on the 
format and content of SASS tabulations were favorable; they also provided suggestions for 
additional tabulation detail. 

In addition to the publication of results from SASS, NCES makes microdata available for 
different or more detailed analyses by users. The public-use data files provided to the public are 
a subset of the full data set. Some variables are suppressed, the level of detail for others is 
reduced because of the need to protect the confidentiality of the respondents. Specifically, 
NCES removes all state identifiers and stratum codes in the public-use files to prevent disclosing 
the identities of schools, individual administrators, and teachers. On the administrator and 
teacher files, income and age are coded into class intervals, and other personal information, such 
as the college or university attended, is converted into an appropriate recode. Detailed affiliation 
codes for private schools are collapsed into three broad categories: Catholic, other religious, and 
nonsectarian schools. School district data in the public-use file cannot be merged with other 
components of the survey. 

The SASS public-use microdata for Rounds 1 to 3 are available on CD-ROM (NCES 98-312), 
which can be obtained by calling Ed Pubs at 1-877-4-EDPUBS, faxing a request to 1-301-470- 
1244, or using the Internet address http://www.ed.gov/pubs/edpubs.html. The CD-ROM also 
includes the data file user's manuals for each round. These manuals provide information on the 
design and procedures for the surveys and include the database documentation and copies of the 
questionnaires. 

Users who need the full data sets for detailed analyses can apply for a license to access the SASS 
restricted-use data. The unabridged data are not subject to the disclosure avoidance procedures 
and, hence, provide a richer database to support detailed analyses. Requests for licensing should 
be sent to the NCES Data Security Office, Statistical Standards Program, NCES/OERI, U.S. 
Department of Education, 1990 K Street, NW, Washington DC 20006. The Data Security Office 





will provide an NCES Restricted-Use Data Manual that describes the procedure to apply for a 
license. 

Another important facet of accessibility is documentation. Clear, concise, complete 
documentation is an essential component of understanding and using SASS data. To assist users, 
whatever their objectives, NCES provides a wide variety of documentation in a number of forms. 
As noted throughout the Profile, a Data File User's Manual is available for each round, both in 
printed form and on CD-ROM, to provide a comprehensive source of information about each of 
the surveys that constitute SASS. In similar fashion, a Sample Design and Estimation Report 
provides a detailed description of the sample design and estimation procedures, including 
variance computation, for each of the surveys in each round. Additional information about 
SASS procedures and data quality is contained in the many SASS methodological publications; 
they may be found on the SASS Home Page (http://nces.ed.gov/surveys/sass), as are SASS data 
and questionnaires. 

The current SASS documentation is primarily cross-sectional in nature, providing factual 
information for each individual round of the survey. This form of documentation is well suited 
to the needs of those who analyze a single round of SASS. Each User's Manual contains a brief 
discussion of changes from the previous round; however it may not fully satisfy the needs of 
those who use 2 or more rounds of SASS data to examine change over time, and of 
methodologists and others who want to understand and assess the evolution of the SASS 
methodology. Analysts of 2 or more rounds of the survey need documentation that clearly draws 
their attention to the definitional and operational changes between rounds that may affect their 
analyses. Methodologists need documentation that provides the rationale for the changes made 
and an assessment of the effectiveness of the changes. As data from more rounds of SASS 
become available, interest in more complete documentation that provides a linkage across rounds 
will increase. 

Timeliness and periodicity. Since the inception of SASS in 1987, significant effort has been 
devoted toward producing the results in a timely fashion. Experience to date indicates that 
steady strides have been made to improve timeliness (see table 9-1). In Round 1, for example, 
information for principals, schools, and teachers was available in about 2 years after the 
completion of data collection; school district information became available in about 3 years. 
Each succeeding round has improved on this timing — data from Round 2 were first published in 
January 1993, about 1 1/2 years after the end of data collection, and Round 3 publication began 
in June 1995, only about 12 months after the end of data collection. Plans for Round 4 call for 
the data to become available even sooner, in spring 2001, only some 8 months following 
completion of data collection. Factors that have contributed to this positive trend of providing 
the data in a shorter time period include: the growing familiarity with the subject matter, which 
leads to standardization and greater efficiency in processing; the repetition of data collection that 
permits investment in technology (such as the use of computer-assisted telephone interviewing, 
advanced data entry, and increased use of computer editing); and improvements in the collection 
procedures and instruments that result in more complete returns and a shorter time period 
required to collect the information. Improvements in data processing also have contributed 
significantly; specifically, capturing data through imaging rather than data keying and using 
generalized software systems throughout the process have led to accelerated processing. 
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Table 9-1. — Timing of core SASS surveys, by round and activity 





Start of 


End of 




First 






data 


data 


Data file 


publication 


Type of 


Round and survey 


collection 


collection 


released 


issued 


publication 



Round 1 (1987-88) 



School district 


1/88 


6/88 


4/91 


8/91 


ED tabs 


Principal 


1/88 


6/88 


4/91 


5/90 


ED tabs 


School 


2/88 


6/88 


4/91 


7/90 


ED tabs 


Teacher 


4/88 


6/88 


4/91 


7/90 


ED tabs 


Round 2 (1990-91) 


School district 


12/90 


6/91 


1/93 


3/93 


Issue brief 


Principal 


12/90 


6/91 


1/93 


6/93 


Issue brief 


School 


12/90 


6/91 


1/93 


6/93 


ED tabs 


Teacher 


2/91 


6/91 


1/93 


6/93 


Issue brief 


Round 3 (1993-94) 


School district 


10/1/93 


6/94 


6/95 


7/95 


ED tabs 


Principal 


10/4/93 


4/94 


6/95 


7/95 


ED tabs 


School 


11/30/93 


5/94 


6/95 


7/95 


: ED tabs 


Teacher 


11/30/93 


6/94 


6/95 


7/95 


ED tabs 



On the issue of periodicity, SASS was designed to provide an ongoing and consistent source of 
data bn the teaching workforce and school population. Rounds 1 to 3 of SASS were conducted 
at 3-year intervals in 1987-88, 1990-91, and 1993-94. The interval between Rounds 3 and 4, 
administered in 1999-00, was extended to 6 years, in part because of budget limitations, thus 
providing an opportunity to reconsider the content and purposes of SASS. 

Some thought has been given to the possibility of a different cycle for SASS. An initial 
exploration of the implications of cycles of varying length was undertaken to provide guidance 
on when the second round of SASS should be conducted. Models were developed to explore the 
tradeoffs between the cost of a survey cycle and the errors of key estimates, with the cost 
depending on periodicity and sample sizes and the error being expressed as a composite of 
sampling error and the error of prediction based on prior year estimates. Several other factors, 
including response burden and the need for time to evaluate the Round 1 content and 
methodology, influenced the decision in favor of a 3-year interval between Rounds 1 and 2. This 
modeling approach is discussed by Ghosh, Kaufman, Smith and Chang (1994), Smith, Ghosh, 
and Chang (1995), and by Smith, Ghosh and Chang (1997) who attempted various optimization 
calculations based on estimates of key variables from the first three rounds. 

The next SASS is currently planned for 2003-04 and is to be conducted on a 4-year cycle 
thereafter, as suggested by a survey of users and discussion by the SASS Technical Review 





Panel and the NCES Advisory Council on Education Statistics (ACES). The following reasons 
support this conclusion: 

■ As a unique source of national and state representative data on important topics in education 
reform, SASS users considered that a 5-year cycle was too long a gap for the survey to 
maintain its currency and provide timely data to support policy planning. 

■ A 4-year cycle beginning with 1999-00 and the next administration in 2003-04 would allow 
SASS to coincide with the cycle of presidential elections and with the reauthorization 
schedule for the major elementary and secondary education legislation. This schedule would 
allow data from SASS to become available around the start of each presidency when the 
government and policymakers need data to inform the planning of new initiatives. 

■ A 4-year cycle for SASS would also allow the possibility of administering the SASS and the 
National Assessment of Educational Progress (NAEP) student assessment at the same time in 
some of the same schools (Skaggs and Kaufman, 2000). A SASS-NAEP linkage is being 
conducted as a research and development project in 1999-00 to enrich the database for 
research. If the linkage is successful and the results prove useful, a similar linkage may be 
sought in future rounds of SASS. The 2-year cycle for NAEP and a 4-year cycle for SASS 
would allow the possibility for the two surveys to be synchronized again in 2003-04. 

9.3 DATA QUALITY 

Based on the detailed findings presented in earlier chapters, this section reviews the main sources 
of potential error for SASS with the aim of identifying areas where methodological 
improvements are most needed and areas where further methodological studies would shed more 
light on data quality issues. The four sources of error discussed below are sampling error, 
coverage error, nonresponse, and measurement error. 

Sampling error. Each of the individual surveys in SASS is designed to produce certain key 
estimates with specified levels of precision. Sample sizes are chosen to satisfy these precision 
requirements. Given this situation, a key issue with regard to sampling error is the efficiency of 
the sample design. 

The assessment of sampling efficiency is complex because of the interrelationships of the several 
survey components and the requirements for estimates of specified precision for many different 
domains. These features inevitably lead to compromises in the sample designs for the individual 
surveys. In particular, since the sample of schools selected for the School Survey is the starting 
point for the samples for all the other surveys, its design places constraints on the sample designs 
for the other surveys. 

The sample design for the School Survey is a compromise design that takes account of the needs 
of both that survey and the Teacher Survey. Schools are sampled with probability proportional 
to a measure of size that is the square root of the number of teachers, as a compromise between 
the equal probability that would be appropriate for the School Survey and probability 
proportional to the number of teachers that would be appropriate for the Teacher Survey (see 
section 2.3). The use of this compromise measure of size also has implications for the other 
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SASS components: sampling schools with equal probability would be more appropriate for the 
Principal Survey, the Library Survey and the Librarian Survey, whereas sampling with 
probability proportional to the number of teachers (which may be roughly proportional to the 
number of students) may be more suitable for the Student Records Survey. In the Teacher 
Demand and Shortage Survey, with a local education agency (LEA) being sampled if any school 
in its district is sampled, the appropriate measure of size for sampling public schools is difficult 
to determine; however, for private schools, sampling with equal probability would be 
appropriate. 

An issue that is related to the choice of a measure of size for sampling schools is the form of the 
estimates to be produced. An equal probability sample of schools is appropriate for the School, 
Principal, Library, and Librarian Surveys if the estimates produced are expressed in terms of 
numbers or percents of schools, principals, libraries, or librarians, with given characteristics, for 
example the percent of principals who consider student use of alcohol a problem in their schools. 
If the problem of alcohol occurs mainly in large schools, the percent of principals considering 
student use of alcohol a problem will be very different frdrh the percent of students in schools 
where the principal considers it a problem, and often the latter percent is the relevant one. In 
discussing this issue, Kish (1965, p. 418) gives the example that, around 1957, one-half of 
American high schools offered no physics, but that these schools accounted for only 2 percent of 
high school students. For most purposes, the 2 percent figure is the more meaningful one. The 
form of estimate to be used is important for choice of sample design since an efficient design for 
student-based estimates would sample schools with probability proportional to the number of 
students, as distinct from the equal probability sampling that is appropriate for school-, principal-, 
library-, or librarian-based estimates. 

Another example of compromise in sample design relates to the. sample allocation used in the 
School Survey to provide domain estimates of specified precision. The smaller Library, 
Librarian, and Student Records Surveys are not designed to provide all these domain estimates, 
and therefore they subsample in a manner that attempts to redress the unequal allocation of the 
sample across domains. However, this subsampling cannot fully compensate for the domain 
oversampling. 

No extensive evaluation of the interrelated sample design for the surveys in SASS has been 
conducted. There are two advantages of the interrelated design: first, data from the different 
surveys can be linked for analysis (for example, data from the principal and teachers in the same 
school can be analyzed together); and, second, there are some cost savings in sample selection. 
However, the interrelated design places a high response burden on sampled schools, which may 
harm response rates, and it involves the compromises discussed above. Since SASS is itself 
evolving and since circumstances are changing, a broad ranging review of the interrelated design 
would be advisable periodically. Such a review could determine whether all the survey 
components should remain interrelated as at present, or whether some of the surveys should be 
conducted separately from the core surveys. 

Assuming that the interrelated design is retained, research could usefully be conducted based on 
data collected in the first 4 rounds to determine whether any improvements in sampling 
efficiency can be obtained. For example, early research led to the decision to sample schools 
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first and then select the LEAs of sampled schools (see section 2.3). This decision could be 
reviewed using the data now available. Also, schools are sampled with probabilities proportional 
to the square roots of their numbers of teachers. The suitability of that measure of size could be 
assessed. A full review of the SASS interrelated sample design would be a complex undertaking 
since many design choices affect different survey components in different ways, but even some 
limited evaluations may lead to useful gains in sample efficiency. 

Coverage error. The ideal sampling frame for a survey would include every element in the 
survey's target population with a single listing for each element. In practice, this ideal is rarely 
achieved, and there is clear evidence that it is not achieved in the component surveys of SASS. 

The issue of school coverage is particularly important in SASS because of the nested structure of 
the surveys. In recent rounds, the Common Core of Data (CCD), supplemented by lists of 
schools from the Bureau of Indian Affairs (BIA) and the Department of Defense (DOD), has 
served as the sampling frame for public schools, and the Private School Survey (PSS), 
supplemented by updated lists of affiliation members, has been used for private schools. In 
Round 4, an additional frame has been included for charter schools. 

Both the CCD and PSS have imperfections as described in section 2.4.3. Since they are used for 
several NCES surveys, their coverage is the subject of broad interest. Several recent studies 
have evaluated their coverage, and continuous efforts will be made to improve them. An issue of 
concern to SASS is that inevitably the frames are out of date for the school year of the survey. 
As a result, new schools and recent school splits and mergers are not reflected on the frames (the 
area frame component of the private school sample partially addresses this issue). It would be 
useful to determine the magnitude of the coverage problem from this source, and also, to 
evaluate the quality of the list of charter schools. 

A significant problem with coverage is that a survey's definition of the units to be covered may 
not conform to the structure and terminology used in different parts of the population. Thus, for 
example, some states consider certain administrative groups of schools to be single schools, 
whereas SASS defines each individual administrative unit to be a school. This kind of problem 
affects both the frame listings and the data reported for a sampled "school." The definitional 
problem arises particularly with students and teachers, since the sampled schools provide the 
listings of these individuals. It is a particularly severe problem in the teacher listing operation, 
since defining who is to be included as a teacher is not straightforward. In this situation, there is 
the risk that the person completing the form will use the school terminology for a teacher rather 
than the SASS definition. 

A particular concern with coverage in the Teacher Survey relates to the operational procedures 
that define the sample. Teachers who are sampled from the teacher listing forms but have left 
the school by the time of the Teacher Survey data collection are treated as out of scope, while 
teachers jpining the school in the interim have no chance of selection. Thus, the survey's 
coverage is neither teachers at the beginning of the school year nor teachers at the time of data 
collection. No study of teacher mobility within a school year has been conducted to date to 
assess the magnitude of the problem. 
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Additional noncoverage problems may . also occur in the School Principal Survey, the Library 
Survey, and the Librarian Survey, where some schools classify themselves as out of scope 
(having no principal, library, or librarian). A study to determine the extent of such classification 
errors would be useful. 

Nonresponse. The response rates to the various SASS surveys have generally been high for 
public schools. For example, in Round 3, the public school response rates for all the surveys that 
were conducted in a single phase of data collection were over 90 percent. The Teacher Survey 
and the Student Records Survey had lower response rates at just over 80 percent as a result of the 
two opportunities for nonresponse. In the Teacher Survey, nonresponse could have occurred 
because the school did not provide the teacher list for sampling teachers or because a sampled 
teacher failed to respond. In the Student Records Survey, nonresponse could have occurred 
because a school did not provide a teacher list and class rosters for sampled teachers or because 
the school failed to return the completed questionnaires for sampled students. The lowest public 
school response rate has been that for the Teacher Followup Survey. Although a high proportion 
of teachers responding to the Teacher Survey respond also to the, Teacher Followup Survey, the 
additional phase of data collection leads to some further losses that resulted in an overall 
response rate of 77 percent for Round 3. 

For private schools, the response rates for all the surveys in SASS have been markedly lower 
than those for public schools. In Round 3, only the School Survey and the Principal Survey had 
response rates of over 80 percent. The response rates for the other surveys were 70 percent or 
somewhat higher, except for the Teacher Followup Survey where the overall response rate was 
only 64 percent. 

As with any repeated survey, continuing attention needs to be given in SASS to maintaining and, 
if possible, increasing response rates. Experimental studies could usefully be conducted to test 
out methods to improve response rates, particularly for the private school components of the 
SASS. A range of possible methods could be considered, including the use of different sponsors 
targeted at different types of schools, the use of incentives, and the use of shorter questionnaires 
that are easier to complete. 

To achieve its final response rates, SASS employs a combination of mail questionnaires followed 
by telephone interviews with mail nonrespondents and field followup, if necessary. The per-unit 
cost of the telephone data collection is much higher than that for mail data collection. Also, 
there are indications that mail responses are of higher quality than telephone responses (although 
this is based on non-experimental data). For both these reasons it is desirable to maximize the 
mail response rates. The use of the postcard reminder cards and allowing a longer interval for 
mail returns in Round 3 may have contributed to higher mail response rates in that round. 
Continued efforts to improve the user-friendly format of the questionnaires and the 
accompanying material may also help to increase mail response rates. 

Item nonresponse rates vary greatly. Many items have high response rates, but there are others 
with low response rates. Some low response rates are likely to result from the difficulty or, in a 
few cases, the sensitivity of the information requested. Others appear to be caused by 
respondents' failure to navigate correctly through a questionnaire's skip instructions. It may be 
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possible to reduce some of those problems by revising the content and wording of questions and 
by changing the format and layout of the questionnaires. 

Recent research on the design of self-completion questionnaires deals with the principles of 
design for navigating the respondent through the questionnaire, as well as more generally for 
obtaining responses of high quality. In addition, advances in printing methods facilitate the use 
of such tools as color, shading, and different font sizes that increase the available design options 
in an important way. Attention to ensuring that the SASS questionnaires are as user-friendly as 
possible addresses not only the item nonresponse problem. It may also reduce total nonresponse, 
obtain more valid responses, and reduce the number of changes made in editing. 

Measurement error. A variety of methods has been used to investigate measurement errors in 
the SASS, including reinterviews, a record check study, in-depth interviews using cognitive 
research techniques, methodological experiments, reviews of completed questionnaires, analysis 
of errors and inconsistencies detected during data processing, and aggregate comparisons of 
survey estimates with estimates from external sources (which deal with all types of error in 
combination). A variety of methods is needed since all methods have their limitations. 

The reinterview program is a core component of the measurement error research in SASS, being 
applied in most of the surveys at each round. This program has been valuable in identifying 
items with high response variance, and many of these items have been revised in later rounds in 
an attempt to reduce the response variance. However, reinterviews have two main limitations. 
First, they measure only inconsistency of response, and thus fail to identify cases where a 
respondent consistently gives a wrong answer (as might occur if the respondent applies his or her 
own understanding of the meaning of a term such as school or teacher, where that understanding 
differs from the survey's definition). Second, by themselves, reinterviews fail to indicate the 
reasons for inconsistency. 

In Round 2, the SASS reinterview program tried to overcome the second limitation by a 
subsequent reconciliation step, using dependent interviewing, but this did not work well. The 
interviewers conducting the reinterviews used a paper-and-pehcil questionnaire on which the 
original responses were recorded on the right-hand side. It seems likely that the interviewers 
were influenced by the original responses in recording the reinterview responses, thereby 
creating the markedly greater consistency observed in the reinterviews in Round 2 than was 
observed in Rounds 1 and 3. A modification of the reinterview procedure that should address 
this problem would be to use CATI for the reinterviews, with the original responses then being 
withheld from the interviewers until the reconciliation part of the interview. Such an approach 
should yield valid estimates of response consistency. Whether the approach is useful depends on 
the ability of the reconciliation process to determine the reasons for the observed inconsistencies, 
with knowledge of those reasons then guiding questionnaire redesign to produce more accurate 
responses. While such reconciliation studies may be useful in SASS, it should be noted that, in 
general, reconciliation studies have had limited success in identifying reasons for inconsistencies 
that could be addressed by questionnaire redesign. 

A common finding from the reinterview program across all surveys has been the low level of 
reliability for opinion questions. This finding is consistent with the results for opinion questions 
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in other surveys. Such unreliability may be acceptable for some limited forms of analysis, but is 
problematic for more detailed analysis. For the latter type of analysis it may be necessary to 
improve the reliability with which a construct is measured by creating an index from the 
responses to several questions relating to that construct. 

Record check studies are often valuable for examining measurement errors, but they also have 
their limitations. Most importantly, they can be used only when the relevant information is 
available on records and access can be obtained. Even when this is the case, there remain 
problems of erroneous matches and failures to match, incorrect information on the records, and 
differences between the definitions of the variable for the records and for the survey. For these 
various reasons, the only record check study conducted in SASS to date has been the teacher 
transcript record check study. 

The attraction of a record check study is that it seeks to determine "true values" with which the 
survey responses can be compared (subject to the limitations indicated above). Another 
approach for obtaining true values is to conduct in-depth followup interviews, with extensive 
questioning and encouragement to respondents to consult records, for a subset of key items (e.g., 
number of full-time equivalent teachers in the school). Not only can this approach give true 
values (with some error), it can also sometimes identify the sources of error (e.g., counting a 
part-time teacher as a full-time teacher). This approach may be useful in a future pilot study 
and/or a future round of SASS. 

Comparisons of SASS estimates with estimates from other sources provide an overall evaluation 
of the SASS estimates. However, the opportunities for such comparisons are very limited, and 
even when they can be made, they tend to be of limited value. Any differences observed may 
reflect definitional differences, differences in the time reference, errors in the other sources, or 
errors in SASS arising from any combination of noncoverage, nonresponse, sampling, 
measurement or processing. As a result, the aggregate comparisons that have been made in 
Rounds 1 to 3 of SASS have been useful in drawing attention to some major discrepancies, but 
have generally not been able to identify the causes of the discrepancies. An extension of the 
aggregate comparison approach is to perform micro-level matching of SASS responses and 
similar data in record sources. This type of match may provide an understanding of the 
discrepancies and, hence, indicate whether changes should be made in SASS. For example, such 
a match conducted at the school level to compare SASS and CCD data on the numbers of full- 
time equivalent teachers found that schools often appeared to report headcounts, rather than FTE 
counts. Application of micro-level matches in other areas could prove equally useful. 

9.4 CONCLUDING REMARKS 

This report has reviewed a variety of error sources including coverage issues, nonresponse, and 
measurement and sampling errors and has provided quantitative measures of error where 
possible. However, in general, the effects that an error source may have on a survey estimate 
cannot be easily quantified. For instance, the lower the response rate, the greater the likelihood 
of a significant nonresponse bias, even after nonresponse adjustments have been made, but the 
magnitude of the bias in a particular estimate is unknown. Furthermore, it has not been feasible 
to combine all the indications of quality into an overall index of total survey error for a given 
survey estimate. Nevertheless, the information on quality presented in the report should help 
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users to decide how much confidence to place in the estimates of interest to them and to 
determine how best to use the survey data in their analyses. 

This report is also intended to help those conducting SASS in directing new survey design efforts 
to areas of the surveys where improvements will be most productive. This chapter suggests a 
number of possible research projects that may guide future methodological developments using 
the current approach. In a broader context, SASS will also need to keep in touch with 
technological advances in communications. In particulzir, the rapid advances taking place in the 
use of the internet suggest that by Round 5 or 6 of SASS the preferred mode of data collection 
may shift from a mail questionnaire to a web-based questionnaire for several of the surveys. A 
number of special resezirch studies will be needed to develop the new methods before such a 
change can be implemented in SASS data collection operations. 

Finally, it should be noted that this edition of the Quality Profile is an update of the earlier 
edition. It is anticipated that the Quality Profile will be updated again in the future to incorporate 
the information about subsequent rounds of SASS. Documentation on the survey operations for 
Round 4 and on research projects is being carefully maintained to assist in the preparation of the 
next edition. Readers Eire invited to submit suggestions for improvements for the next edition to: 
SASS Quality Profile, 1990 K Street, N.W., Washington, D.C. 20006. 
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